INC_03020207 – History MV fails after adding new column to source PySpark MV

[saiteja1_cdns]

We are using a PySpark script inside Incorta to call an API and load data into a Materialized View (full load). Since loading this MV directly into a table is not feasible, we store the data in a history Materialized View (incremental mode) to maintain snapshots.
When a new column is added to the source MV, the full load completes successfully; however, the incremental load of the history MV fails with INC_03020207 (schema mismatch). Please advise on the recommended approach to handle schema changes in this setup without losing historical data.

[dylanwan]

If you have the historical data produced within the MV, and cannot be reloaded from the source, you can first backup the data using another MV in a full load and modify the MV logic to load from the backup MV with the new column.  I assume that you have a logic to deal with the new column for the historical data without going back to the source.  

After the full load that migrate the data from the backup MV, you can change it to a regular full load and you can start accumulate new incremental data.

Hope this helps

Let us know if you need help

Thanks,

Dylan