Where is my heap going?

Elastic Stack Elasticsearch
1

I have a heap of size 23 GB and here is the breakdown of different components in the heap as showed by the node stats API,

Field-data Cache = 529 MB
Filter Cache = 2.8 GB
Query Cache = 0
ID Cache = 0
Segments in Mem = 228 MB
Index Writer Mem = 31 MB
Version Map = 30 KB
Translog Size = 6 MB
Fixed Bitset = 7.2 MB
Current Merges = 219 MB

Clearly, sum of all the components listed above is not equal to the 23 GB. I'm curious to know where is my heap actually getting consumed. We often have few hosts in the cluster which have high heap utilization than others. We use bulk API to index and search also happen on the same cluster.

During the time of sampling, there are 34 search contexts open and 10351 file descriptors on the hosts. Search and bulk request queues were also 0.We have increased the bulk threads to 96 (we know we should limit it to number of cores, we plan to do it but not sure if it has anything to do with this).

1 Like
2

Hey,

Elasticsearch claims all the configured heap on startup - but this does not mean it is going to be used. When you find out that much less heap is used, you should assign less memory for Elasticsearch. First you will have more memory available for the file system cache, second garbage collections that clear out a lot of memory will run faster, because there is less memory to clear.

--Alex

3

If you have configured 23 GB for the heap, the Java VM will reserve 23 GB at start time, no matter if used or not.

The node stats are the node stats, not the JVM heap stats. They can not be understood as components in the heap. They are estimations of byte memory regions, either referenced on the heap or off-heap or even on disk (translog), maybe already gone or recycled in buffer areas.

Use JDK tools like jmap for heap dumps, which are expensive, or use tools like visualvm to watch live JVM heap stats. There you can see the consumptions.

If you are not sure, then please, don't play with the bulk thread pool settings. Increasing makes situation worse than intended, meaning more memory consumption, more segments, more file descriptors, more lagging behind.

4

@jprante @spinscale Thanks for your responses. I guess there was some confusion created with that statement, I meant that nodeStats shows 23 GB of JVM heap space is consumed overall, I have given 30 GB of space.

So this JVM heap space shown by nodeStats cannot be mapped back to just the components shown in the other parts of the nodeStats API response like field-data, filter caches etc, right ?

Since we use bulk indexing APIs and I have read that each bulk request is loaded in the memory of the node, is there an API to check the amount of heap taken up by such requests ?

I'm basically looking forward to collect information about each and every byte of the JVM heap space (if possible...) and taking a heap dump in production can be risky and complex to analyze, so are there other APIs that can show more detailed output or maybe any other component of the heap that can contribute majorly and is not part of node stats. We usually see high memory pressure on our nodes, reaching near to 90% of the heap memory on the normal days even. I want to make sure we are not wasting unnecessary memory anywhere.

Another peculiar observation is that it reaches 90% on few nodes at a time, though this set/number of hosts is not constant. Is there any reason behind this ?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.