0

Incremental load from Data Lake

I have data loading from Azure Data Lake Storage (ADLS) using Data Lake connection in Incorta, when I enable incremental load for this table, how does Incorta find the incremental data ? I do not see any additional place holder for defining incremental load as we get when data is loaded from Oracle tables.

5replies Oldest first
  • Oldest first
  • Newest first
  • Active threads
  • Popular
  • When you have the wildcard union option turned on (that is, you want all the files under a single directory to be included in the table), incremental load will be based on the files last modified timestamp. For example, if you start with a directory that contains 5 files, the first full load event will extract all 5 files, the next incremental load will look for files with last modified timestamp more recent than the last load event time. This includes new files added to the directory and previously existing files that have been modified after the last time they were loaded.

    When you have the wildcard union option turned off (that is, you want a single file to be included in the table), you will have an option to provide an update file. So, only the first file will be extracted in the full load events, and only the the update file will be extracted in the incremental load event.

    Usually in a data lake setup, the first option is more convenient and better suits business use cases.

    Like
    • Ahmed Moawad Thanks Ahmed, it explains exactly what I needed and agree first option seems more suitable for us too.

      Like
  • Ahmed Moawad , I was chatting with Mukul and he showed me how they were bringing in the flat file from ADLS. What if the flat file is saved on local directory? How do we achieve incremental loading?

    Like
    • Swapna Chennam same approach. You will find the same options in data lake data sets as well as local file system data sets. 

      Like
    • Ahmed Moawad I did exactly what  you mentioned  and I received this error - "INC_004028001:Internal IO exception [null]"

      Like
Like Follow
  • 1 mth agoLast active
  • 5Replies
  • 35Views
  • 3 Following