Incremental Loading - how do you use it after 5.1?

mike_mascitti — Fri, 11 Nov 2022 20:30:26 GMT

Hello - we are in the process of upgrading from 5.0 to 5.2 and have noticed a new requirement in 5.1. Changing a 'Key' column in a physical table now requires a full load of that table.

We use incremental loads to load daily snapshots from source systems, and we incrementally load datasets for time-series analysis. Incremental loading also works well with slowly changing dimensions because you can load only changes and let the table build over time.

From my testing, I've found this new requirement to be true for any column, not only a 'Key'. Adding a new measure or dimension will also trigger this warning. (Removing a column does not trigger the warning, but if you remove it and put it back then it does.)

In 5.2, if you add a new column and do not perform a Full Load you will receive an 'Index out of bounds' error on any data selection that includes the new column.

This is a significant restriction to the way incremental loading works. In version 5.0, adding a new column works as expected. The column shows up with null values for prior loads and is populated from when it was added. This even works for adding new keys.

What is the company philosophy for tables loaded incrementally? They provide tremendous value and it's made Incorta the best way for us to easily view day-over-day changes of datasets. As users perform analysis adding new columns is a normal practice, but losing all your history in the process will usually be unacceptable.

Does anyone know of a workaround for this? Attached are some screenshots of the warnings.

Re: Incremental Loading - how do you use it after 5.1?

BrandonR — Tue, 15 Nov 2022 22:02:25 GMT

This isn't a great workaround, but I do remember a similar problem being addressed and a workaround given here:

https://community.incorta.com/t5/data-schema-discussions/schema-full-load-after-changes/m-p/823

topic Incremental Loading - how do you use it after 5.1? in Data & Schema Discussions

Incremental Loading - how do you use it after 5.1?

Re: Incremental Loading - how do you use it after 5.1?