I have a two-node ELK cluster, each having about 250G RAM, and ES_HEAP_SIZE is set to 32G, with below circuit breaker settings:
the cluster recently hit exceptions:
ElasticsearchException CircuitBreakingException[[parent] Data too large, data for [message] would be larger than limit of [23911392870/22.2gb]
and followed with exceptions like:
background thread had an uncaught exception
ElasticsearchException[failed to flush exporter bulks]
Suppressed: ElasticsearchException[failed to flush [default_local] exporter bulk] nested: ElasticsearchException[failure in bulk execution:
: index [.marvel-es-2016.04.13], type [node_stats]
and finally the out of memeory exception
failed engine [out of memory (source: [create])] java.lang.OutOfMemoryError: Java heap space
I googled a bit on this error, and there are two recommendations:
- add a new node in
- increase heap size
for option 2, since the heap size is already set to 32gb, in consideration of pointer address byes (4 vs 8 bytes), should I still increase ES_HEAP_SIZE? or actually option 1 is the only choice for this case?
And is this caused by the user query from Kibana (too much return data)? or something else?
Thanks a lot in advance!