What happends when heap_used_percent hits 100%?

Elastic Stack Elasticsearch
1

Hello, ElasticSearch users.
I'm very new to ElasticSearch and just publish an ElasticSearch cluster.

My first try, without setting ES_HEAP_SIZE, gave me OutOfMemory Exception and the cluster lost all the nodes.

Now I've set ES_HEAP_SIZE to 50% of total memory (as recommended) and I am currently monitoring node stats.

It looks like heap_used_percent grows very quickly. It hit 70% in 6 hours.

So I am worrying about another OOM.

Could anyone tell me what happends when heap_used_percent hits 100%?

Thanks.

2

I found a document on it.

It says,

The heap_used_percent metric is a useful number to keep an eye on. Elasticsearch is configured to initiate GCs when the heap reaches 75% full. If your node is consistently >= 75%, your node is experiencing memory pressure. This is a warning sign that slow GCs may be in your near future.

If the heap usage is consistently >=85%, you are in trouble. Heaps over 90–95% are in risk of horrible performance with long 10–30s GCs at best, and out-of-memory (OOM) exceptions at worst.

3

When it hits 100% it'll usually OOM.
If it is growing too fast then you need to increase the heap, add other nodes to the cluster, or reduce the amount of data in the cluster.

1 Like