cancel
Showing results for 
Search instead for 
Did you mean: 

Transformation error in Pyspark Query

msinha8
Ranger

Hello Gurs,

I am new to trying to write Pyspark query in incorta to filter a data for last week (Week starts from Monday and Ends on Sunday) 

But I am getting Transformation error (error log attached) while trying to save the query <<query attached>> 

also, I am observing that left join is not working as expected.

i.e. 

When filtering the data for a specific date (from table B), only those data is getting displayed (from  table A) for which data is available even though table B is left joined with table A.

 

Dataset Type : Materialized View

Language : Spark Python

 

 

 

 

2 REPLIES 2

anurag
Employee
Employee

@msinha8 is there a reason you are using MV for filtering data by last week?

Few other approaches you can consider are:

1. Create analyzer table with filter by last one week 

2. Filter by last week in dashboard using applied filter

Thanks @anurag , I was able to complete the assignment with approach# 1

Regards,

Mritunjay Sinha