topic Re: MV Incremental Load in Data & Schema Discussions

MV Incremental Load

kxie — Mon, 19 Sep 2022 19:35:21 GMT

Can anyone confirm that get_last_refresh_time() that is referenced in the link below is meant to be equivalent to "?" for SQL MV's? This function only returns 0 for me.

https://community.incorta.com/t5/dashboards-analytics/creating-materialized-view-using-pyspark/ta-p/734

Re: MV Incremental Load

dylanwan — Mon, 19 Sep 2022 20:01:52 GMT

It is a python function and can be used in PySpark as the example shown in the document.

For SQL MV, please use "?".

Re: MV Incremental Load

kxie — Mon, 19 Sep 2022 20:08:46 GMT

I'm looking for the PySpark equivalent to ?. Is get_last_refresh_time() meant to be equivalent to that in a Pyspark MV? I am looking to implement incremental loads in a Python MV and need to filter the tables based on the last refresh time.

Re: MV Incremental Load

dylanwan — Mon, 19 Sep 2022 22:51:26 GMT

Which release off incorta are you using? The question mark may work in pyspark as well but the last refresh time fuction was the equivalent to question mark. You will need to pass the value to the filter as shown in the example using % to replace it.

Another way is to use the max value in a separate query and use that query to filter the data.

In a more recent release, the max value will be available from incorta loader using the ? replacement and you don't need to get the value by yourselves.