Schedule job for load of individual tables

It seems that Parquet actually captures all of the same rows again in an incremental load but only updates/deleted based on Key in the in-memory database. I have a table that does not have an ideal way of performing incremental loads and wanted to see if it's possible to schedule a full load of that table only in the schema.

11replies Oldest first
  • Oldest first
  • Newest first
  • Active threads
  • Popular
  • Currently, it is not supported to schedule table loads selectively. A scheduled load applied to the whole schema only. A workaround is to place such tables in a separate schema. An ER already exist for such a request but there is no due date yet.

    • Ahmed Esmat   Is there an update of when this feature is going to be available ?

    • Sunita Datta  Not considered yet

  • Is this option available now ?

    • Gnaneshwar Muthaluri There is no schedule job that can be created for a single table.   A schedule job has to be created for a schema, as of the current version, Incorta 4.8.x

      However, when a scheduled incremental refresh is specified as an Incremental Load, Incorta will execute the Incremental extraction logic when the incremental logic exists.  Incorta will execute the full extract logic during incremental refresh when the incremental logic is not defined.  

      If you have a source table that is refreshed by truncate and reload, or there is no clear way to define a key, by not entering the incremental logic, the table will be extracted and reload in every incremental refresh.

    • Amit Kothari Dylan Wan  If a given schema is scheduled as an Incremental Load and some tables in the schema are not set up to load incrementally, those specific tables will be loaded as full refresh (truncate and reload). Please confirm.

    • Sunita Datta Yes, your understanding is correct.

    • Dylan Wan Thank you.

    • I hope Incorta product / engineering team take this customer need actively. Due to the limitation of scheduling  group/ individual tables in a schema on different data refresh timings, we are having to create multiple schemas unnecessarily. The main concern is on the managment / maintenance of such schemas.

  • If i use the CLI I am able to load a single table as full load but I am not able to load as incremental. Is that something available in any new version ?

    • Gnaneshwar Muthaluri  incremental load is always at a schema level.

Like1 Follow
  • 1 yr agoLast active
  • 11Replies
  • 226Views
  • 7 Following

Product Announcement

A new community experience is coming! If you would like to have beta access to provide feedback, please contact us at community@incorta.com.