Elasticsearch stopped working

Hello Team

Unfortunately, my elasticsearch stopped working. and I'm not sure what to do. Please have a look at my logs

org.elasticsearch.action.NoShardAvailableActionException: No shard available for [get [.kibana][doc][config:6.2.3]: routing [null]]

org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed

[ERROR][o.e.x.w.e.ExecutionService] [EC2AMAZ-1763048] failed to update watch record [ArTI0TAdSnGq1KU4pC80rA_logstash_version_mismatch_24a9faf8-c860-4201-9618-e848f723cbda-2018-04-18T09:41:49.266Z]
org.elasticsearch.ElasticsearchTimeoutException: java.util.concurrent.TimeoutException: Timeout waiting for task.

org.elasticsearch.transport.ReceiveTimeoutTransportException:

fatal error in thread [elasticsearch[EC2AMAZ-1763048][bulk][T#2]], exiting
java.lang.OutOfMemoryError: Java heap space.

Not sure what all these errors/warnings are? Also all my indices turned red.

I'm indexing a small file less than 20mb. Still not sure why this happens?

is it because of bulk index queue setting?

Please do give some suggestions on this. what are the next steps that I need to do to avoid this?

Thanks for your time as always :slight_smile:

This is the key error from your logs:

This means that your JVM does not have enough memory to continue, you need to increase the heap size, decrease the usage, or add more nodes to your cluster so that the load can be spread around.

Once the server is up I would also recommend checking the /_nodes/stats API endpoint to try and determine what is using the heap memory, so you can adjust accordingly.c

Thank you @dakrone.
I'm indexing 12mb log file which has over 50000 logs. Still not sure why this is happening.
My jvm.options file:

-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+AlwaysPreTouch
-Xss1m
-Djava.awt.headless=true
-Dfile.encoding=UTF-8
-Djna.nosys=true
-XX:-OmitStackTraceInFastThrow
-Dio.netty.noUnsafe=true
-Dio.netty.noKeySetOptimization=true
-Dio.netty.recycler.maxCapacityPerThread=0
-Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true
-Djava.io.tmpdir=${ES_TMPDIR}
-XX:+HeapDumpOnOutOfMemoryError
-Xmx4096m
-Xms4096m

As you said I've also seen the /_nodes/stats api.

"jvm" : {
"timestamp" : 1524113669615,
"uptime_in_millis" : 441428,
"mem" : {
"heap_used_in_bytes" : 793517296,
"heap_used_percent" : 18,
"heap_committed_in_bytes" : 4277534720,
"heap_max_in_bytes" : 4277534720,
"non_heap_used_in_bytes" : 135423072,
"non_heap_committed_in_bytes" : 144363520,
"pools" : {
"young" : {
"used_in_bytes" : 39239880,
"max_in_bytes" : 139591680,
"peak_used_in_bytes" : 139591680,
"peak_max_in_bytes" : 139591680
},
"survivor" : {
"used_in_bytes" : 15543264,
"max_in_bytes" : 17432576,
"peak_used_in_bytes" : 17432576,
"peak_max_in_bytes" : 17432576
},
"old" : {
"used_in_bytes" : 738734152,
"max_in_bytes" : 4120510464,
"peak_used_in_bytes" : 738734152,
"peak_max_in_bytes" : 4120510464
}
}
}
but not able to understand clearly.

Please suggest me next steps to solve this? Do I need to increase the heap memory? If so how much?
Thanks for your time as always :slight_smile:

Hi @rahulnama,

Change the elasticsearch jvm.options to 1gb like following it should work for you.
-Xmx1g
-Xms1g

Thanks & Regards,
Krunal.

Hey, Thanks for your response, @Krunal_kalaria.

I think in my jvm.options heap memory is 4GB and you are suggesting me to reduce it to 1GB. The error I got is Heap out of memory. Quite confusing! Is my understanding correct?

1 Like

Try to reduce coz i was facing the same issue that time i was reduce and then its worked for me.

You got an OOM. Increase your Heap size from 4 to 6 and see how it handles the results, if the same then increase to 8GB for min and max. Please note to ensure you have the same RAM available for the OS and its file handles.

1 Like

Thank you much for your suggestion @JKhondhu. Anyway, My system RAM is 8GB. So as per blog post you have suggested I shouldn't go beyond 4GB of maximum heap size right?

It depends on your use case. If you have a 12mb log file (daily) that sounds relatively not much at all actually.

What did you see on the OS logs?

4GB sounds suffice for getting going but if you are hitting OOM you will need to increase your heap space.

Sure Thanks @JKhondhu. Not sure about the os logs : ( . If possible please let me know how to debug.
and one more thing I have doubt is

When I created Indices based on date I'm ending up with all the issues which is stopping elasticsearch. but when I create only one index no issues and everything is fine? Please do suggest me why is this happening?

You will need to be reviewing the elasticsearch.log before during and after you saw this fatal logging:
fatal error in thread [elasticsearch[EC2AMAZ-1763048][bulk][T#2]], exiting java.lang.OutOfMemoryError: Java heap space.

How are you creating daily indices? Logstash?
How is data being indexed?

What troubleshooting have you done up until now?

I don't see why daily indices would be causing issues here unless your cluster is relatively too small to be coping with the amount of data you wish to onboard?

@JKhondhu

Yes I'm using Logstash for creating the indices and I'm data is being indexed based on time stamp.

Till now, I tried with different Heap memory settings frequently checking the logs. but that didn't help me anyway.

when I created multiple indices based on timestamp many shards created which I think caused OOM.
what helped me is creating a single index which ended up with creating very few shards comparatively.

Still Not sure why many shards are creating when I create multiple indices.

Thanks :slight_smile:

The default number of shards is 5P and 1R.

You can modify this in your mappings or templates. This just may be the case of ensuring all your new indices are of a lower number of shards for example. 1P, 0R.

https://www.elastic.co/guide/en/elasticsearch/reference/6.2/indices-templates.html
https://www.elastic.co/guide/en/elasticsearch/reference/6.2/indices-update-settings.html
https://www.elastic.co/guide/en/elasticsearch/reference/5.5/index-modules.html#dynamic-index-settings

Perhaps you need to know your retention period and delete indices older than X days old so your small cluster can cope - or grow it out, vertically or horizontally to fit your needs!

I will go through the documentation as you suggested and also through mappings and templates.

Thanks for all your time. so helpful :slight_smile: @JKhondhu

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.