What happends when heap_used_percent hits 100%?


#1

Hello, ElasticSearch users.
I'm very new to ElasticSearch and just publish an ElasticSearch cluster.

My first try, without setting ES_HEAP_SIZE, gave me OutOfMemory Exception and the cluster lost all the nodes.

Now I've set ES_HEAP_SIZE to 50% of total memory (as recommended) and I am currently monitoring node stats.

It looks like heap_used_percent grows very quickly. It hit 70% in 6 hours.

So I am worrying about another OOM.

Could anyone tell me what happends when heap_used_percent hits 100%?

Thanks.


#2

I found a document on it.

https://www.elastic.co/guide/en/elasticsearch/guide/current/_monitoring_individual_nodes.html#_jvm_section

It says,

The heap_used_percent metric is a useful number to keep an eye on. Elasticsearch is configured to initiate GCs when the heap reaches 75% full. If your node is consistently >= 75%, your node is experiencing memory pressure. This is a warning sign that slow GCs may be in your near future.

If the heap usage is consistently >=85%, you are in trouble. Heaps over 90–95% are in risk of horrible performance with long 10–30s GCs at best, and out-of-memory (OOM) exceptions at worst.


(Mark Walkom) #3

When it hits 100% it'll usually OOM.
If it is growing too fast then you need to increase the heap, add other nodes to the cluster, or reduce the amount of data in the cluster.


(system) #4