Elastic warm node heap unexpected behavior

[farin99 (Yoni Farin)]

Hi,

One of our elasticsearch clusters is showing an unexpected behavior on the heap of its warm nodes. The cluster consists of 40 hot data nodes (8 cores, 64gb ram out of which 30.5gb heap and 1.5tb of local SSD storage), and 30 warm data nodes (8 cores, 32gb ram out of which 16gb heap and 3tb of attached HDD storage).

Briefly, it looks like the node is accumulating some kind of temporary objects (query caching / shard aggregation data / inverted indices etc.), eventually fills up all the heap, and then suddenly drops some of this temporary data and clearing several GBs. It is easily evident in the following 7-day query of the heap size of one of the machines:

The temporary object loading policy and GC policy creating this behavior is not clear to us, and we are sure they can (and should) be optimized.
We'd like to check what data is used to fill up the heap, what elasticsearch parameters control the heap loading policy of this data type and what optimizations can be done to get a more consistent behavior of each of the nodes.

For reference, here is a previous discussion about our warm storage's heap:

Choosing the right heap size for warm data nodes

Hi One of our production clusters has 40 hot data nodes (8 cores, 64gb ram out of which 30.5gb heap and 1.5tb of local SSD storage), and 30 warm data nodes (8 cores, 32gb ram out of which 16gb heap and 5tb of attached HDD storage). Recently, as the warm data nodes have been filled up with more data and crossed the 1.6tb of used storage, we have begun seeing an interesting pattern: the java heap usage circulates around 14-16gb, CPU is around 15% and a lot of time is invested in JVM GC collection…

Any help is appreciated.

Thanks.