Transformation error in Pyspark Query

[msinha8]

Hello Gurs,

I am new to trying to write Pyspark query in incorta to filter a data for last week (Week starts from Monday and Ends on Sunday) 

But I am getting Transformation error (error log attached) while trying to save the query <<query attached>> 

also, I am observing that left join is not working as expected.

i.e. 

When filtering the data for a specific date (from table B), only those data is getting displayed (from  table A) for which data is available even though table B is left joined with table A.

Dataset Type : Materialized View

Language : Spark Python

[anurag · Accepted solution]

@msinha8 is there a reason you are using MV for filtering data by last week?

Few other approaches you can consider are:

1. Create analyzer table with filter by last one week 

2. Filter by last week in dashboard using applied filter

[msinha8]

Thanks @anurag , I was able to complete the assignment with approach# 1

Regards,

Mritunjay Sinha