Elasticsearch service keeps failing

Elastic Stack Elasticsearch
1

Hi All,

I have implemented EFK in my environment. Kibana and Elasticsearch are installed on a separate instance and Fluentd is configured on my kubernetes cluster. Everything was working perfect. But sometimes my Elasticsearch servicec keeps failing. When I checked elasticsearch log I got following error

[2018-09-11T03:58:59,801][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [] fatal error in thread [Thread-19], exiting
java.lang.OutOfMemoryError: Java heap space

My Kibana instance is having 25 GB memory. Seeking help?

2

You're running out of heap space for some reason.

What version are you on? What OS? What JVM? How many nodes? How many shards/indices? How many GB is that?

3

1)Elasticsearch verion: 6.4.0
2)OS: Ubuntu 16.04.5
3)JVM option: -Xms1g -Xmx1g
4)Please let me know how can I check the node Number. In my elasticsearch.yml i do not found the number.

5)For indices Please find the result from below:

root@kibana-1:/etc/elasticsearch# curl 'localhost:9200/_cat/indices?v'
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open logstash-2018.08.31 L1Yqf1YGR4m0kqeudgs_4Q 5 1 45 0 124.3kb 124.3kb
yellow open logstash-2018.09.06 TWpq_84_RsK0CTGJSnsYAA 5 1 9 0 67.7kb 67.7kb
yellow open logstash-2018.09.04 ddWM_M_bSWGFNh2ZAr6RRQ 5 1 13 0 69.8kb 69.8kb
yellow open logstash-2018.08.29 ON16PQ5qQrqOwAyNLNLyFw 5 1 93 0 295.5kb 295.5kb
green open .kibana t8OQX5PnRDSPomkws-BxBw 1 0 2 0 13.5kb 13.5kb
yellow open logstash-2018.09.02 i62EaffxQs219A7huOWiZw 5 1 41 0 84.8kb 84.8kb
yellow open logstash-2018.09.11 pO0U_NXfSjS6YRDJ11BoGw 5 1 2975551 0 412.9mb 412.9mb
yellow open logstash-2018.09.10 MMLf5kY0QuqYogmoSRmg4Q 5 1 40556237 0 8.3gb 8.3gb
yellow open logstash-2018.08.28 AVIT2NqIQpm9HZ_Prd759w 5 1 104 0 115kb 115kb
yellow open logstash-2018.08.30 qShOtVXvRcGha6n8NG2-SQ 5 1 41 0 84.9kb 84.9kb
yellow open logstash-2018.09.05 KjLglQhRSgukkhzNA5S2YA 5 1 28 0 76.7kb 76.7kb
yellow open logstash-2018.09.01 ZAHwckWLSg-NZ8sE8PB10Q 5 1 42 0 84.9kb 84.9kb
yellow open logstash-2018.08.24 bJ8Bf6OXRYuMGlK3OmFW4A 5 1 2198639 0 520.7mb 520.7mb
yellow open logstash-2018.08.25 qDwDVtDnRg-L1xIFipE1Uw 5 1 604211 0 229.1mb 229.1mb
yellow open logstash-2018.08.26 OJDPmRAJSIy0nNlDS9i9qQ 5 1 392863 0 141.8mb 141.8mb

I have integrated my three cluster enviornment's fluentd to a single elasticsearch and a large amout of indices files are generating. Do not know whether this is the reason.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.