topic Re: Dense Snapshot Truncation? in Data & Schema Discussions

Dense Snapshot Truncation?

marcpaige — Mon, 01 Aug 2022 17:37:24 GMT

I've read and implemented a dense snapshot table. It works exactly as expected. Yay!

What I need now is a way to trim the snapshot. The start of the snapshot has a starting date way back when. I want to move that starting date forward so as to limit the range of the data without losing any data.

tia!

Re: Dense Snapshot Truncation?

RADSr — Mon, 01 Aug 2022 22:51:41 GMT

I think - if I read the question correctly - that the only way to accomplish this is to run an insight into .csv, edit the .csv, and then reload.

AFAIK there is no way to edit the actual parquet file directly.

Re: Dense Snapshot Truncation?

marcpaige — Tue, 02 Aug 2022 01:28:28 GMT

I get it. The data source for the snapshot is a SQL database. I don't understand how to make the switch. I get the export and edit. Reloading means changing the data source. If I change the data source back from text file to database will it work?

Re: Dense Snapshot Truncation?

dylanwan — Tue, 02 Aug 2022 08:47:03 GMT

No, there is no supported way to manipulate the parquet files from Incorta.

If your concern is about the size of the data that will take the memory space in the Incorta engine, you may add a Data Load filter and run a load from staging.

Otherwise, I must say that I did not try this myself, but we may query the data from another MV code to back up the data, and optional apply a filter during querying the snapshot table ti build the backup.

The data in the original snapshot table will be empty when we run a full load, but a second data source that uses an Incorta over Incorta SQL (with postgreSQL connector) can run against the backup MV to bring those old snapshot data back. The incremental refresh from the original data set can continue building the the new snapshots. The 2nd data set that brings the data from the backup can have a 1=0 filter to return nothing during incremental.
Periodically when the old snapshot data have to be removed, run the above processes as full load manually at the table level.

Re: Dense Snapshot Truncation?

Incortaconsulta — Wed, 17 Aug 2022 17:19:52 GMT

Hi - Wondering, whether there is a Video clip or Incorta page screenshots on how to create snapshots. That would be easier to perceive than a text document.

Re: Dense Snapshot Truncation?

RADSr — Wed, 17 Aug 2022 17:26:16 GMT

This isn't a video, but offers a step-by-step:

https://community.incorta.com/t5/data-schemas-knowledgebase/creating-snapshot-tables-with-incorta/ta-p/811