topic Re: Update ( add columns to ) a snapshot table in Administrative Discussions

Update ( add columns to ) a snapshot table

RADSr — Mon, 13 Nov 2023 20:19:32 GMT

So you've built yourself a snapshot table and things are going along swimmingly... and then BAM! you want to add another column to it.

You can't do a full reload because all that sweet history would just disappear and never by properly appreciated. Unfortunately altering the Incorta table structure *requires* a full load.

What I want though is the DB equivalent of altering the table and putting default values in for NULLs ( although really what I'll be doing is manufacturing approximate data from context specific history to populate those fields ).

I'm sure I can do this in a series of tables using the original snapshot as a source, but that means every time I make a change I have to create another, new, snapshot table.

I turn to you fellow Community members... how would you approach such a conundrum?

Re: Update ( add columns to ) a snapshot table

mhelmy — Tue, 14 Nov 2023 10:53:31 GMT

Hello RADSr,

We are nearing the completion of a new feature designed to support schema evolution, allowing for the seamless addition or removal of columns. This enhancement directly addresses your request to eliminate the need for a full load after making changes to the schema.

With this new functionality, actions such as adding or deleting columns will no longer necessitate a full load or rewriting of the parquet files.

Re: Update ( add columns to ) a snapshot table

RADSr — Tue, 14 Nov 2023 14:13:09 GMT

Thank you @mhelmy - is this a "take it easy for the holidays and expect this as a New Year's present" or a "find your workaround now and it'll be easier later" thing? 😉

Re: Update ( add columns to ) a snapshot table

mhelmy — Tue, 21 Nov 2023 13:54:37 GMT

Expected in Q1 2024.

Re: Update ( add columns to ) a snapshot table

shoaibshaik — Fri, 03 May 2024 23:02:24 GMT

Is this Feature added to the application @mhelmy