ES_HEAP_SIZE above 32g?

Elastic Stack Elasticsearch
1

Hi,

I have a two-node ELK cluster, each having about 250G RAM, and ES_HEAP_SIZE is set to 32G, with below circuit breaker settings:
indices.fielddata.cache.size: 75%
indices.breaker.fielddata.limit: 85%

the cluster recently hit exceptions:
ElasticsearchException CircuitBreakingException[[parent] Data too large, data for [message] would be larger than limit of [23911392870/22.2gb]

and followed with exceptions like:
background thread had an uncaught exception
ElasticsearchException[failed to flush exporter bulks]
at org.elasticsearch.marvel.agent.exporter.ExportBulk$Compound.flush(ExportBulk.java:104)
at org.elasticsearch.marvel.agent.exporter.ExportBulk.close(ExportBulk.java:53)
at org.elasticsearch.marvel.agent.AgentService$ExportingWorker.run(AgentService.java:201)
at java.lang.Thread.run(Thread.java:745)
Suppressed: ElasticsearchException[failed to flush [default_local] exporter bulk] nested: ElasticsearchException[failure in bulk execution:
[0]: index [.marvel-es-2016.04.13], type [node_stats]

and finally the out of memeory exception
failed engine [out of memory (source: [create])] java.lang.OutOfMemoryError: Java heap space

I googled a bit on this error, and there are two recommendations:

  1. add a new node in
  2. increase heap size
    for option 2, since the heap size is already set to 32gb, in consideration of pointer address byes (4 vs 8 bytes), should I still increase ES_HEAP_SIZE? or actually option 1 is the only choice for this case?

And is this caused by the user query from Kibana (too much return data)? or something else?

Thanks a lot in advance!

2

Look at adding a few more ES processes on the same nodes rather than increasing the heap on the existing ones.

Otherwise if you hit GC it will kill things.

1 Like
3

Hi Mark, thank you very much for replying.
"adding a few more ES processes on the same node", are you referring to running multiple nodes on the same machine?

And is this problem caused by the user query from Kibana (too much return data)? or something else?

4

Yes.

Partially. What version are you on?

5

Hi. If you have large servers you can create more than 2-3 ES instances in the same time. Look at the example https://www.elastic.co/elasticon/conf/2016/sf/scaling-elasticsearch-at-netsuite-good-bad-and-ugly , they have 5 ES instances on one physical server.

1 Like
6

I'm on version 2.2.1

7

thank you, I'll definitely watch it!