topic Re: Incremental load last load time precision in Administrative Discussions

Incremental load last load time precision

RADSr — Wed, 19 Apr 2023 21:56:16 GMT

I have a table w/ incremental loading enabled using " <timestampcolumn> > ? "

Troublingly, I have some duplicate rows with the same <timestampcolumn> values ( and all other values ). I'm not sure why, but the load time was the same time ( w/ in 4 seconds ) of the <timestampcolumn>

I see in the documentation there is a lookback setting for some sources - although I don't see that configuration in my SQL Database source.

If that's configurable, where? And/or what is the precision of "?" for the last load time.

Re: Incremental load last load time precision

amit_kothari — Fri, 21 Apr 2023 15:02:02 GMT

? is the epoch time stored in the time log file for that table in our server. Duplicate rows could be due to key columns as Incorta does a upsert based on the keys so can you check the key fields?

Re: Incremental load last load time precision

RADSr — Mon, 24 Apr 2023 17:52:44 GMT

1) Kudos to support who spent three hours on the phone w/ me trying to track down my issue!

2) As @amit_kothari mentions above, the "?" is stored in epoch time

3) It's precise to the millisecond

4) My specific mystery gets more and more interesting. The source table timestamp showed up in a file produced before the timestamp indicates the record was produced.

5) According to the source system support folks that timestamp is based on the time a record is uploaded.

6) The obvious next check would be time zone offsets, but there aren't enough records duplicated to make that likely. Nevertheless, it is on my list!

So I've found the precision is to the millisecond.

Re: Incremental load last load time precision

JoeM — Tue, 25 Apr 2023 13:38:35 GMT

Lots of great info here @RADSr - thanks for sharing your findings with the community!