topic Re: Chaining of schema loads in Data & Schema Discussions

Chaining of schema loads

sushant_lkq — Sat, 22 Oct 2022 07:14:29 GMT

Hi Everyone,

We have this specific requirement where we want to create a workflow for schema loads. For example Schema B load should happen only after schema A is loaded and so on. As a workaround we can setup appropriate schedule timing for each schema but offcourse it won't guarantee this behaviour .

Has anyone faced and achieved this in Incorta?

Thanks for your support!

Regards,

Sushant

Re: Chaining of schema loads

Prince — Tue, 25 Oct 2022 12:35:02 GMT

Hi @sushant_lkq ,

It depends on which version of Incorta you have?

For on-prem/older releases, you can use the CLI python scripts.

For coming/recent/cloud versions, there are public APIs are currently in Beta, which can help to integrate with some scripts or tools like AirFlow for example.

And later on, there's a feature on the roadmap, that would allow scheduling multiple schemas from the scheduler, and it will respect the dependency between them.

Which Incorta version do you have?

Re: Chaining of schema loads

RADSr — Wed, 26 Oct 2022 20:43:25 GMT

Here's an idea ( and yes, it's a kludge 😞

1) Go to the CMC and set the loader to fail on first error.

2) Change the load order of your schema so that a "test" table is first and in its own group

3) Write test table so it fails if the predecessor load hasn't completed for that day -- I don't know the details here, but it looks like you could call a stored proc based on the Incorta metadata load and force that fail if the last load date <> current_date or something like that

4) Schedule the schema load to run every n-minutes. If the first table fails, the load will not continue, if it does not fail it will. If you were really ambitious you could include timestamps in the logic to force the test table to fail if it had already run successfully that day or w/in whatever cadence you are aiming for.

Or wait for the roadmaped enhancements 😉

But if you do fiddle around w/ this idea please post back as I'm curious to see if it could work.

Re: Chaining of schema loads

sushant_lkq — Thu, 03 Nov 2022 13:25:23 GMT

Thanks for the information. We are on Incorta version 5.1.5 as of now.

Re: Chaining of schema loads

kxie — Thu, 03 Nov 2022 20:27:45 GMT

Do you have any documentation on how to do this with python scripts in CLI?