Elasticsearch service keeps failing

Hi All,

I have implemented EFK in my environment. Kibana and Elasticsearch are installed on a separate instance and Fluentd is configured on my kubernetes cluster. Everything was working perfect. But sometimes my Elasticsearch servicec keeps failing. When I checked elasticsearch log I got following error

[2018-09-11T03:58:59,801][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [] fatal error in thread [Thread-19], exiting
java.lang.OutOfMemoryError: Java heap space

My Kibana instance is having 25 GB memory. Seeking help?

You're running out of heap space for some reason.

What version are you on? What OS? What JVM? How many nodes? How many shards/indices? How many GB is that?

1)Elasticsearch verion: 6.4.0
2)OS: Ubuntu 16.04.5
3)JVM option: -Xms1g -Xmx1g
4)Please let me know how can I check the node Number. In my elasticsearch.yml i do not found the number.

5)For indices Please find the result from below:

root@kibana-1:/etc/elasticsearch# curl 'localhost:9200/_cat/indices?v'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open logstash-2018.08.31 L1Yqf1YGR4m0kqeudgs_4Q 5 1 45 0 124.3kb 124.3kb
yellow open logstash-2018.09.06 TWpq_84_RsK0CTGJSnsYAA 5 1 9 0 67.7kb 67.7kb
yellow open logstash-2018.09.04 ddWM_M_bSWGFNh2ZAr6RRQ 5 1 13 0 69.8kb 69.8kb
yellow open logstash-2018.08.29 ON16PQ5qQrqOwAyNLNLyFw 5 1 93 0 295.5kb 295.5kb
green open .kibana t8OQX5PnRDSPomkws-BxBw 1 0 2 0 13.5kb 13.5kb
yellow open logstash-2018.09.02 i62EaffxQs219A7huOWiZw 5 1 41 0 84.8kb 84.8kb
yellow open logstash-2018.09.11 pO0U_NXfSjS6YRDJ11BoGw 5 1 2975551 0 412.9mb 412.9mb
yellow open logstash-2018.09.10 MMLf5kY0QuqYogmoSRmg4Q 5 1 40556237 0 8.3gb 8.3gb
yellow open logstash-2018.08.28 AVIT2NqIQpm9HZ_Prd759w 5 1 104 0 115kb 115kb
yellow open logstash-2018.08.30 qShOtVXvRcGha6n8NG2-SQ 5 1 41 0 84.9kb 84.9kb
yellow open logstash-2018.09.05 KjLglQhRSgukkhzNA5S2YA 5 1 28 0 76.7kb 76.7kb
yellow open logstash-2018.09.01 ZAHwckWLSg-NZ8sE8PB10Q 5 1 42 0 84.9kb 84.9kb
yellow open logstash-2018.08.24 bJ8Bf6OXRYuMGlK3OmFW4A 5 1 2198639 0 520.7mb 520.7mb
yellow open logstash-2018.08.25 qDwDVtDnRg-L1xIFipE1Uw 5 1 604211 0 229.1mb 229.1mb
yellow open logstash-2018.08.26 OJDPmRAJSIy0nNlDS9i9qQ 5 1 392863 0 141.8mb 141.8mb

I have integrated my three cluster enviornment's fluentd to a single elasticsearch and a large amout of indices files are generating. Do not know whether this is the reason.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.