on 04-27-2022 10:20 AM
Most of the "rules of thumb" we apply for sizing our Incorta environment depend on the amount of compressed data we expect in-memory, which we call "x" below. For now, the easiest way to calculate x is to use the number showing in the Incorta Schema page (or the sum for all Tenants/Schemas). The list below enumerates the main rules we apply when sizing our Incorta hardware.
Incorta v4 supports both on-heap and off-heap memory management for both the analytics and the loader service. On-heap memory refers to memory managed inside the java process itself, or the java "heap" space. Off-heap memory refers to memory managed outside the java process. As java processes, the analytics service and loader service both have settings in the CMC that control the on-heap usage vs. the off-heap usage. Please see the screenshot below from the CMC.
As a rule of thumb, it is a good practice to keep about 70-75% of the memory offered to Incorta off-heap. This keeps the amount of memory in the java process for the loader and analytics services (aka. the on-heap memory) smaller which makes java maintenance activities like garbage collection less intrusive.
If we revisit the concept of "x" again from the "Rules of Thumb" section above, we are ultimately interested in the amount of compressed data we expect in the Incorta storage layer (aka. "x"). Based on the rules of thumb above, we will plan for 2x memory for the loader service (2.2x for v4.3) and 1x memory for the analytics service. The "Memory Size" setting controls the on-heap memory size and this should be 20-25% of the 2x or 1x memory size, for the loader and analytics service respectively. The off-heap component should be set to 70-75% of the 2x or 1x memory size, for the loader and analytics service respectively.
Keep in mind these settings only control the amount of memory allocated to the Incorta services. For example, you may have a 1TB RAM server and only choose to allocate 200GB to the loader service and 100GB to the analytics service.