.png "Employee"){kind=icon}
Horizontal Scaling
Enterprises require data and analytics solutions that can not only handle massive amounts of data loads frequently but also scale seamlessly to meet evolving business user needs for concurrent analytics. Incorta is designed from the ground up to address these challenges with a focus on enterprise-grade scalability, ensuring that your data platform grows with you.
This document will focus on Incorta’s capabilities to scale and provide the high performance needs in an optimal and efficient manner.
Incorta's Scalability Pillars
Incorta's architecture is fundamentally built for massive scale, resting on two key pillars:
1. Incorta Loader
At the core of most Incorta customers deployments is a high frequency refresh of data from multiple systems of record and disparate source systems. This results in multiple load plans being refreshed, at times concurrently. It involves data acquisition, data enrichments using Spark and the load and post processing activities.
Data Acquisition and Data Loads
- Scalable Loader Workers for Effortless, Massively Parallel Data Ingestion, harness a fleet of dynamically scaling loader workers that spin up on demand to load tens and hundreds of tables simultaneously.
- Unified Support for Scheduled Batch Loads and Always-On Streaming Pipelines
- Intelligent Resource Optimization and Built-In High Availability
Scalable Transformation or Enrichment Engine
- Code-first, serverless Spark, Incorta’s Spark-on-Kubernetes Chidori workload management spins up and tears down the right executors on demand, so you customers can focus on logic, while Incorta addresses the infrastructure needs at peak demand spikes.
- Run multiple container images side-by-side in the same cluster—one job with Python 3.11 + pandas, the next with Spark ML + custom JARs, the third with R tidyverse.
- Fine-grained autoscaling expands from a handful of CPUs to thousands in seconds, matching the burstiness of heavy ETL, feature-engineering, or model-training pipelines.
2. Incorta Analytics
Incorta analytics utilizes advanced memory management techniques to keep the most widely used columns loaded into memory and readily available, drastically reducing I/O latency. The platform is engineered to efficiently handle thousands of concurrent queries without degradation in performance. However, for customers who have a planned or predictable surge in concurrent users or complex dashboards usage in a short period of time, Incorta leverages a distributed, scale-out architecture that allows you to increase analytics capacity by simply adding more analytical services. The primary driver for scale is query processing driven by user concurrency or query complexity.
- With Incorta’s elastic architecture, Analytics Engines scales instantly based on Queries predictive lifetime - no additional warehouses to manage.
- Schedule Scaling based on workload to optimize consumption cost with different Analytics Engines sizes from (XSmall into 6X Large)
- Define your Min and Max number of Analytics Engines and with Autoscaling Incorta Smart Workload Management starts and stops Engines as needed to dynamically manage the workload
This horizontal scaling capability means that if you need to support a sudden increase in users, you can quickly and cost-effectively expand the cluster without any downtime or complex configuration changes.
Scale up and scale down
Scale up can be initiated as a one time scale up or scheduled via a scheduler like Google Cloud Scheduler. There is a coordinated down time the first time the scalable infrastructure is put in place, which warrants analytics services restart. Subsequent scale ups and scale downs will not require any further downtime. All active running queries will continue executing till the new service/s are up and running.
Session Management
- User distribution follows a consistent hashing mechanism to minimize disruption.
- Existing sessions generally remain on their original service until they end naturally.
- Only a small percentage of active sessions may be reassigned.
Next Steps: Optimizing Your Incorta Scalability
To ensure your Incorta deployment is optimally configured to meet both current and future data load and analytical demands, we encourage you to leverage the built-in scalability features.
If you anticipate a significant, planned surge in data volume, load frequency, or concurrent user activity that may necessitate scaling your Incorta Loader or Analytics processes, please reach out to your Account Executive or Customer Success Manager.
They can help you:
- Review current performance metrics and growth projections.
- Model the necessary scale-up or scale-out adjustments.
- Schedule any necessary coordinated infrastructure changes to minimize impact.
Proactive planning ensures that Incorta continues to deliver sub-second performance and seamless scalability as your enterprise needs evolve.
