topic Re: Garbage Collection in Administrative Discussions

Garbage Collection

Incortaconsulta — Mon, 13 Feb 2023 17:45:29 GMT

I Have 500 Gbbox , where I have assigned 365 Gb off-heap and 125 Gb on-heap memory, but while loading a table I m getting error message with the following.

- INC_03010102: Not enough memory while extracting table GC overhead limit exceeded. Please contact administrator to increase ON heap space.

Kindly suggest me suitable config for the above memory.

Re: Garbage Collection

JeffW — Mon, 13 Feb 2023 21:02:15 GMT

Ram, a little more information please? 365+125 = 490 on a box having 500 limit. So, please tell me a little more? Are you Cloud or On-Prem? If On-Prem, how many servers do you run? Analytics? Loader? Spark? CMC? MySQL? Zookeeper? On which servers do all of these critical components of Incorta run? Understanding your installed physical architecture will assist in answering your question.

Jeff

Re: Garbage Collection

Incortaconsulta — Tue, 14 Feb 2023 06:45:15 GMT

Hi Jeff - It's on-prem.

analytics = 500Gb( Zookeeper, spark,postgres, CMC)

loader = 500 Gb (Spark process I can see).

Re: Garbage Collection

JeffW — Wed, 15 Feb 2023 20:08:22 GMT

How long has the On / Off Heap been in this config sizing = 490GB out of 500? This is too high for these boxes and does not seem to be leaving much room for OS and other processes to run. And if you are relying on Spark to load some of your data, I see no room allowed for Spark itself to run if it is shared on Loader box, or is it running on Analytics? Have you opened a ticket for this, because it seems that you generally have too much memory allocated to the services and are not leaving enough room for other processes.

Re: Garbage Collection

ahmedezzeldeen — Sat, 18 Feb 2023 19:02:43 GMT

Can you open a ticket for this. 125 GB seems to be enough for OnHeap memory. It strange with that onheap memory to have GC overhead error. The error generally means that OnHeap needs to be increased or decrease the concurrency of schema loading.