topic Re: Transformation error in Pyspark Query in Dashboards & Analytics Discussions

Transformation error in Pyspark Query

msinha8 — Fri, 01 Sep 2023 20:36:38 GMT

Hello Gurs,

I am new to trying to write Pyspark query in incorta to filter a data for last week (Week starts from Monday and Ends on Sunday)

But I am getting Transformation error (error log attached) while trying to save the query <<query attached>>

also, I am observing that left join is not working as expected.

i.e.

When filtering the data for a specific date (from table B), only those data is getting displayed (from table A) for which data is available even though table B is left joined with table A.

Dataset Type : Materialized View

Language : Spark Python

Re: Transformation error in Pyspark Query

anurag — Tue, 05 Sep 2023 14:50:27 GMT

@msinha8 is there a reason you are using MV for filtering data by last week?

Few other approaches you can consider are:

1. Create analyzer table with filter by last one week

2. Filter by last week in dashboard using applied filter

Re: Transformation error in Pyspark Query

msinha8 — Tue, 12 Sep 2023 23:42:59 GMT

Thanks @anurag , I was able to complete the assignment with approach# 1

Regards,

Mritunjay Sinha