Chaining of schema loads

[sushant_lkq]

Hi Everyone,

We have this specific requirement where we want to create a workflow for schema loads. For example Schema B load should happen only after schema A is loaded and so on. As a workaround we can setup appropriate schedule timing for each schema but offcourse it won't guarantee this behaviour .

Has anyone faced and achieved this in Incorta?

Thanks for your support!

Regards,

Sushant

[Prince]

Hi @sushant_lkq ,

It depends on which version of Incorta you have?

For on-prem/older releases, you can use the CLI python scripts.

For coming/recent/cloud versions, there are public APIs are currently in Beta, which can help to integrate with some scripts or tools like AirFlow for example.

And later on, there's a feature on the roadmap, that would allow scheduling multiple schemas from the scheduler, and it will respect the dependency between them.

Which Incorta version do you have?

[sushant_lkq]

Thanks for the information. We are on Incorta version 5.1.5 as of now.

[kxie]

Do you have any documentation on how to do this with python scripts in CLI?

[RADSr]

Here's an idea ( and yes, it's a kludge 😞

1) Go to the CMC and set the loader to fail on first error. 

2) Change the load order of your schema so that a "test" table is first and in its own group

3) Write test table so it fails if the predecessor load hasn't completed for that day  --  I don't know the details here, but it looks like you could call a stored proc based on the Incorta metadata load and force that fail if the last load date <> current_date or something like that 

4) Schedule the schema load to run every n-minutes.   If the first table fails, the load will not continue, if it does not fail it will.    If you were really ambitious you could include timestamps in the logic to force the test table to fail if it had already run successfully that day or w/in whatever cadence you are aiming for.   

Or wait for the roadmaped enhancements     😉 

But if you do fiddle around w/ this idea please post back as I'm curious to see if it could work.