So you've built yourself a snapshot table and things are going along swimmingly... and then BAM! you want to add another column to it.
You can't do a full reload because all that sweet history would just disappear and never by properly appreciated. Unfortunately altering the Incorta table structure *requires* a full load.
What I want though is the DB equivalent of altering the table and putting default values in for NULLs ( although really what I'll be doing is manufacturing approximate data from context specific history to populate those fields ).
I'm sure I can do this in a series of tables using the original snapshot as a source, but that means every time I make a change I have to create another, new, snapshot table.
I turn to you fellow Community members... how would you approach such a conundrum?
We are nearing the completion of a new feature designed to support schema evolution, allowing for the seamless addition or removal of columns. This enhancement directly addresses your request to eliminate the need for a full load after making changes to the schema.
With this new functionality, actions such as adding or deleting columns will no longer necessitate a full load or rewriting of the parquet files.