Do I need the "where" clause on an incremental update?

RADSr — Thu, 06 Jul 2023 18:33:40 GMT

This is on my list of things to test, but it's a long list 😉

If I have keyed my table w/ the business keys + last update date ( AsOfDate ) and I just copy/paste my query into the "incremental query" section of the data source it should still work as intended, right?

Any new records would be added including "new" records based on the last update date.

What the "where" clause buys me is a potentially faster load-time because I'd be querying a small subset of all the data.

I *think* the cost of doing it w/out a where clause - in addition to longer load-times - might be larger parquet files for each incremental update? But I'm not completely sure how or if that's a showstopper.

Thoughts?

Re: Do I need the "where" clause on an incremental update?

dylanwan — Wed, 31 Jan 2024 19:24:19 GMT

Extracting the same data during incremental refresh will add the cost to the process of merging the records. Incorta extract will insert all data fetched to the file system and thus requires disk I/O. The data will be merged during compaction and will be loaded to Incorta loader service as an UPSERT. Reducing the volume of data being processed will reduce the cost and improve the performance.

topic Re: Do I need the "where" clause on an incremental update? in Administrative Discussions

Do I need the "where" clause on an incremental update?

Re: Do I need the "where" clause on an incremental update?