Schema full load after changes

[sathyan75]

I have a table that I am loading from Salesforce. Salesforce purges the deleted records after 15 days. However, I would like to retain these records in Incorta. Whenever a change is made to the underlying salesforce object, and we include those fields in the salesforce table, Incorta is throwing an error as it is expecting to have a full reload of the table. Is there a workaround for this?

[sathyan75]

Hi RADSr, thanks for that. 

1) I am not aware of how many new columns are going to be added. It keeps happening and changing.

2) Same here, I cant keep creating new tables for every change that is happening.

[RADSr]

I hear you.  The only alternative I see is building the select with a high number of dummy columns and mapping as new source columns come on board.  

Or - and I'm kind of old so still believe in databases  😉  - create a staging table on your RDBMS to which you can add nullable columns and use that as your archive, then do full loads using that as your source.   

That way when columns were added you'd only need to re-validate and add the new columns to Incorta w/ the old rows having NULL values and the new rows having their proper values.

Depending upon how often new columns will be coming on board you can stay pretty simple ( "oh no, something failed"** and go fix ) or go complex ( ETL scripting checking table definitions for changes, writing DDL to add columns, etc.   ).

** I don't think anything will fail if you explicitly select columns v. select *  so you may need a more comprehensive test than "oh no, something failed"  😉    )