cancel
Showing results for 
Search instead for 
Did you mean: 

Incremental load last load time precision

RADSr
Captain
Captain

I have a table w/  incremental loading enabled using " <timestampcolumn> > ? " 

Troublingly, I have some duplicate rows with the same <timestampcolumn> values ( and all other values ).    I'm not sure why, but the load time was the same time ( w/ in 4 seconds ) of the <timestampcolumn> 

I see in the documentation there is a lookback setting for some sources - although I don't see that configuration in my SQL Database source.

If that's configurable, where?  And/or what is the precision of "?" for the last load time.

-- IncortaOne@PMsquare.com --
3 REPLIES 3

amit_kothari
Employee
Employee

? is the epoch time stored in the time log file for that table in our server. Duplicate rows could be due to key columns as Incorta does a upsert based on the keys so can you check the key fields?

RADSr
Captain
Captain

So

1) Kudos to support who spent three hours on the phone w/ me trying to track down my issue!

2) As @amit_kothari  mentions above, the "?" is stored in epoch time

3) It's precise to the millisecond

4) My specific mystery gets more and more interesting.   The source table timestamp showed up in a file produced before the timestamp indicates the record was produced.

5) According to the source system support folks that timestamp is based on the time a record is uploaded.

6) The obvious next check would be time zone offsets, but there aren't enough records duplicated to make that likely.   Nevertheless, it is on my list!  

So I've found the precision is to the millisecond.   

-- IncortaOne@PMsquare.com --

JoeM
Community Manager
Community Manager

Lots of great info here @RADSr - thanks for sharing your findings with the community!