04-19-2023 02:56 PM
I have a table w/ incremental loading enabled using " <timestampcolumn> > ? "
Troublingly, I have some duplicate rows with the same <timestampcolumn> values ( and all other values ). I'm not sure why, but the load time was the same time ( w/ in 4 seconds ) of the <timestampcolumn>
I see in the documentation there is a lookback setting for some sources - although I don't see that configuration in my SQL Database source.
If that's configurable, where? And/or what is the precision of "?" for the last load time.
Solved! Go to Solution.
04-21-2023 08:02 AM
? is the epoch time stored in the time log file for that table in our server. Duplicate rows could be due to key columns as Incorta does a upsert based on the keys so can you check the key fields?
04-24-2023 10:52 AM
So
1) Kudos to support who spent three hours on the phone w/ me trying to track down my issue!
2) As @amit_kothari mentions above, the "?" is stored in epoch time
3) It's precise to the millisecond
4) My specific mystery gets more and more interesting. The source table timestamp showed up in a file produced before the timestamp indicates the record was produced.
5) According to the source system support folks that timestamp is based on the time a record is uploaded.
6) The obvious next check would be time zone offsets, but there aren't enough records duplicated to make that likely. Nevertheless, it is on my list!
So I've found the precision is to the millisecond.
04-25-2023 06:38 AM
Lots of great info here @RADSr - thanks for sharing your findings with the community!