Enhancements to Incorta's Loader Service Architect...

mhelmy · ‎08-31-2023

Introduction

Incorta has made significant architecture enhancements to its loader service, starting with version 2022.11.0, resulting in improved loader performance and increased resilience. In this article, we will highlight the key improvements introduced in both cloud and on-premises releases.

Architecture Enhancements in 2022.11.0

The 2022.11.0 release introduced a new architecture for the loader service, bringing several noteworthy changes. The load job now consists of a planning phase, a merged phase for extraction and enrichment, and modified load and post-load phases.

Planning Phase

During the planning phase, Incorta now effectively plans load jobs and identifies dependencies between tables, materialized views (MVs), formulas, and joins. This allows for concurrent execution of table extraction and MVs transformation, eliminating the need to wait for all tables to be extracted before running MVs. Additionally, post-load calculations are no longer executed in separate phases. By utilizing a dependency graph generated as a directed acyclic graph (DAG), Incorta optimizes the load job by eliminating unnecessary waiting time and limitations in post-load calculations. Memory utilization is also improved, as Incorta reads column data into memory just before it is needed rather than reading all columns before the post-load calculations commence. These enhancements optimize the load job and pave the way for additional improvements and increased control over loading jobs.

Cloud Release 2023.1.0

In the cloud release 2023.1.0, Incorta introduced the preview feature of multi-schema schedules (Load Plans) to replace single schema loading jobs. This feature allows customers to create load plans consisting of multiple schemas, enabling multiple schemas to be loaded within a single load job. Loading multiple schemas in a single load job brings several benefits. First, post-load calculations are optimized by calculating cross-schema joins once within the same load job. Second, the number of load plans is reduced, simplifying load plan administration for schema managers, particularly when dealing with many schemas. For example, only one load plan is needed instead of requiring eight separate load plans to load Oracle EBS standard dimensions incrementally. However, it is important to note that this approach may exert pressure on the loader's memory if the schemas being loaded simultaneously require more memory than what is allocated to the loader. To address this issue, Incorta made necessary adjustments in version 2023.4.0.

Enhancements in 2023.4.0 and Incorta On-premises 6.0

In the 2023.4.0 release and Incorta On-premises 6.0 release, Incorta focused on enhancing off-heap memory management within the loader service. These enhancements enable the loader to handle memory starvation issues that may arise when loading large schemas simultaneously. Additionally, retry and pause mechanisms were introduced for memory-bound tasks, enhancing the loader's resilience by extending the load time of a load job instead of failing it when facing off-heap memory starvation.

Sequential Load Groups in 2023.7.0

In the latest release, 2023.7.0, Incorta extended the functionality of load plans to support sequential load groups. This means that a single load plan can now have multiple sequential load groups of schemas. Time-based scheduling is no longer the sole method for scheduling schema loads. Using load groups allows schema managers to handle cases where schemas need to run one after the other or when they want to distribute the load evenly throughout the day without the need for time buffers between schema loads using time-based load plans. For instance, you can efficiently handle a scenario where one schema contains an Incorta table and another schema contains an MV that reads from that Incorta table by creating a load plan with two load groups, placing the first schema in the first load group and the second schema in the second load group.

The addition of sequential load groups is planned for an on-premises release, scheduled as version 6.1, and expected to be available by the end of the year.

Conclusion

Incorta has made significant architecture enhancements to its loader service, available in both on-premises (version 6.0 and above) and cloud environments (version 2022.11.0 and above). These improvements have resulted in optimized loader performance and increased resilience. Introducing features like multi-schema schedules, enhanced memory management, and support for sequential load groups will give schema managers greater control and efficiency in their data-loading processes. Incorta continues to innovate and improve its offerings, ensuring a seamless experience for its customers.