Huge memory leaks caused by ElasticSearch

Elastic Stack Elasticsearch
1

I have recently taken on the task of managing our existing ES index and in doing so had to alter the index in a way to allow sorting on certain fields. Ever since my updates to the index mapping and an increase in total traffic I/O we have been seeing more and more memory leaks where we could go from 3GB/28GB to 27GB/28GB in less than an hour even though a top command reveals ES only taking up 14% of the total memory.

Here is our configuration:

1 node w/ 28GB RAM on Azure VM (Ubuntu)
1 Index w/ 83.4GB of data

What other information would you like to see to help debug? Thanks!

2

Are you talking about heap or system memory?

3

System Memory (via 'top' command'), is there a way for me to query the actual heap size in real time?

4

Top tends to display memory used for disk caching. Have a look at free. The top line includes the caching and the second line is memory used by running programs. Both are in use - elasticsearch relies on the disk cache to make its job fast. But the cache is only cleared when full so its fairly normal to see it fill up and stay full.

You can see heap size in realtime with jstat or the jvm node stats from elasticsearch. Keep in mind that heap is pretty similar to the cache - its allocated over time and only freed when it gets close to full. So the instantaneous heap values aren't all that interesting. You really need to graph heap and look at the space between the GCs.

All and all, unless you are seeing a problem its not worth getting worked up over.