Integrate Incorta Analyst Studio Scheduler with Incorta Load Plan Scheduler

[treyscheid]

For dashboards and pipelines which use both Analyst studio flows, and incorta tables, it is a necessity to orchestrate runs and loads together. We are looking for ability to add both into one type of schedule or plan in a unified UI (ideally the load plan scheduler we depend on today). The new scheduler should have the same rich features: sequential group, intra group dependencies, optional stop on failure and status notifications.

Potential solution designs I propose:

1. A new "type" property you decide for your first and each subsequent load group as ("schema" or "flow") then you can add multiple of that type. If it is possible to map dependencies for flows and tables together, then this is unnecessary.

2. Add the option to execute a flow by running a full load in Incorta schema on the output table. Either all nodes in the flow, or only the nodes relevant to a specific incorta output table). This would allow orchestrating flow outputs in incorta load plan without scheduler changes. (Since some flows have many outputs this may not work for all orchestration use cases)

While that full UI integration is in development, new public API endpoints to control the Analyst Studio Scheduler would satisfy my needs. In that case a pyspark MV may be used in a standard load plan to start, then poll status for an analyst studio execution. This would counter some of the benefits of processing speed and low overhead that comes with using Analyst studio.