cancel
Showing results for 
Search instead for 
Did you mean: 

MV Incremental Load

kxie
Ranger

Can anyone confirm that get_last_refresh_time() that is referenced in the link below is meant to be equivalent to "?" for SQL MV's? This function only returns 0 for me. 

https://community.incorta.com/t5/dashboards-analytics/creating-materialized-view-using-pyspark/ta-p/...

3 REPLIES 3

dylanwan
Employee
Employee

It is a python function and can be used in PySpark as the example shown in the document.  

For SQL MV, please use "?".

I'm looking for the PySpark equivalent to ?. Is get_last_refresh_time() meant to be equivalent to that in a Pyspark MV? I am looking to implement incremental loads in a Python MV and need to filter the tables based on the last refresh time.

dylanwan
Employee
Employee

Which release off incorta are you using?  The question mark may work in pyspark as well but the last refresh time fuction was the equivalent to question mark. You will need to pass the value to the filter as shown in the example using % to replace it.

Another way is to use the max value in a separate query and use that query to filter the data.

In a more recent release, the max value will be available from incorta loader using the ? replacement and you don't need to get the value by yourselves.