Elasticsearch stopped working

rahulnama · April 18, 2018, 9:59am

Hello Team

Unfortunately, my elasticsearch stopped working. and I'm not sure what to do. Please have a look at my logs

org.elasticsearch.action.NoShardAvailableActionException: No shard available for [get [.kibana][doc][config:6.2.3]: routing [null]]

org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed

[ERROR][o.e.x.w.e.ExecutionService] [EC2AMAZ-1763048] failed to update watch record [ArTI0TAdSnGq1KU4pC80rA_logstash_version_mismatch_24a9faf8-c860-4201-9618-e848f723cbda-2018-04-18T09:41:49.266Z]
org.elasticsearch.ElasticsearchTimeoutException: java.util.concurrent.TimeoutException: Timeout waiting for task.

org.elasticsearch.transport.ReceiveTimeoutTransportException:

fatal error in thread [elasticsearch[EC2AMAZ-1763048][bulk][T#2]], exiting
java.lang.OutOfMemoryError: Java heap space.

Not sure what all these errors/warnings are? Also all my indices turned red.

I'm indexing a small file less than 20mb. Still not sure why this happens?

is it because of bulk index queue setting?

Please do give some suggestions on this. what are the next steps that I need to do to avoid this?

Thanks for your time as always

dakrone · April 18, 2018, 4:55pm

This is the key error from your logs:

This means that your JVM does not have enough memory to continue, you need to increase the heap size, decrease the usage, or add more nodes to your cluster so that the load can be spread around.

Once the server is up I would also recommend checking the /_nodes/stats API endpoint to try and determine what is using the heap memory, so you can adjust accordingly.c

rahulnama · April 19, 2018, 5:04am

Thank you @dakrone.
I'm indexing 12mb log file which has over 50000 logs. Still not sure why this is happening.
My jvm.options file:

-XX:+UseConcMarkSweepGC
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+AlwaysPreTouch
-Xss1m
-Djava.awt.headless=true
-Dfile.encoding=UTF-8
-Djna.nosys=true
-XX:-OmitStackTraceInFastThrow
-Dio.netty.noUnsafe=true
-Dio.netty.noKeySetOptimization=true
-Dio.netty.recycler.maxCapacityPerThread=0
-Dlog4j.shutdownHookEnabled=false
-Dlog4j2.disable.jmx=true
-Djava.io.tmpdir=${ES_TMPDIR}
-XX:+HeapDumpOnOutOfMemoryError
-Xmx4096m
-Xms4096m

As you said I've also seen the /_nodes/stats api.

"jvm" : {
"timestamp" : 1524113669615,
"uptime_in_millis" : 441428,
"mem" : {
"heap_used_in_bytes" : 793517296,
"heap_used_percent" : 18,
"heap_committed_in_bytes" : 4277534720,
"heap_max_in_bytes" : 4277534720,
"non_heap_used_in_bytes" : 135423072,
"non_heap_committed_in_bytes" : 144363520,
"pools" : {
"young" : {
"used_in_bytes" : 39239880,
"max_in_bytes" : 139591680,
"peak_used_in_bytes" : 139591680,
"peak_max_in_bytes" : 139591680
},
"survivor" : {
"used_in_bytes" : 15543264,
"max_in_bytes" : 17432576,
"peak_used_in_bytes" : 17432576,
"peak_max_in_bytes" : 17432576
},
"old" : {
"used_in_bytes" : 738734152,
"max_in_bytes" : 4120510464,
"peak_used_in_bytes" : 738734152,
"peak_max_in_bytes" : 4120510464
}
}
}
but not able to understand clearly.

Please suggest me next steps to solve this? Do I need to increase the heap memory? If so how much?
Thanks for your time as always

Krunal_kalaria · April 19, 2018, 6:13am

Hi @rahulnama,

Change the elasticsearch jvm.options to 1gb like following it should work for you.
-Xmx1g
-Xms1g

Thanks & Regards,
Krunal.

rahulnama · April 19, 2018, 6:57am

Hey, Thanks for your response, @Krunal_kalaria.

I think in my jvm.options heap memory is 4GB and you are suggesting me to reduce it to 1GB. The error I got is Heap out of memory. Quite confusing! Is my understanding correct?

Krunal_kalaria · April 19, 2018, 8:27am

Try to reduce coz i was facing the same issue that time i was reduce and then its worked for me.

JKhondhu · April 19, 2018, 8:51am

You got an OOM. Increase your Heap size from 4 to 6 and see how it handles the results, if the same then increase to 8GB for min and max. Please note to ensure you have the same RAM available for the OS and its file handles.

rahulnama · April 19, 2018, 9:07am

Thank you much for your suggestion @JKhondhu. Anyway, My system RAM is 8GB. So as per blog post you have suggested I shouldn't go beyond 4GB of maximum heap size right?

JKhondhu · April 19, 2018, 9:52am

It depends on your use case. If you have a 12mb log file (daily) that sounds relatively not much at all actually.

What did you see on the OS logs?

4GB sounds suffice for getting going but if you are hitting OOM you will need to increase your heap space.

rahulnama · April 19, 2018, 1:06pm

Sure Thanks @JKhondhu. Not sure about the os logs : ( . If possible please let me know how to debug.
and one more thing I have doubt is

When I created Indices based on date I'm ending up with all the issues which is stopping elasticsearch. but when I create only one index no issues and everything is fine? Please do suggest me why is this happening?

JKhondhu · April 19, 2018, 1:45pm

You will need to be reviewing the elasticsearch.log before during and after you saw this fatal logging:
fatal error in thread [elasticsearch[EC2AMAZ-1763048][bulk][T#2]], exiting java.lang.OutOfMemoryError: Java heap space.

How are you creating daily indices? Logstash?
How is data being indexed?

What troubleshooting have you done up until now?

I don't see why daily indices would be causing issues here unless your cluster is relatively too small to be coping with the amount of data you wish to onboard?

rahulnama · April 19, 2018, 1:54pm

@JKhondhu

Yes I'm using Logstash for creating the indices and I'm data is being indexed based on time stamp.

Till now, I tried with different Heap memory settings frequently checking the logs. but that didn't help me anyway.

when I created multiple indices based on timestamp many shards created which I think caused OOM.
what helped me is creating a single index which ended up with creating very few shards comparatively.

Still Not sure why many shards are creating when I create multiple indices.

Thanks

JKhondhu · April 19, 2018, 2:02pm

The default number of shards is 5P and 1R.

You can modify this in your mappings or templates. This just may be the case of ensuring all your new indices are of a lower number of shards for example. 1P, 0R.

https://www.elastic.co/guide/en/elasticsearch/reference/6.2/indices-templates.html
https://www.elastic.co/guide/en/elasticsearch/reference/6.2/indices-update-settings.html
https://www.elastic.co/guide/en/elasticsearch/reference/5.5/index-modules.html#dynamic-index-settings

Perhaps you need to know your retention period and delete indices older than X days old so your small cluster can cope - or grow it out, vertically or horizontally to fit your needs!

rahulnama · April 20, 2018, 6:38am

I will go through the documentation as you suggested and also through mappings and templates.

Thanks for all your time. so helpful @JKhondhu

system · May 18, 2018, 6:38am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Elasticsearch stops working, says failed to update shard information Elasticsearch	5	5628	August 21, 2019
Searchphaseexecutionexception Elasticsearch	4	360	July 9, 2019
org.elasticsearch.action.UnavailableShardsException: [ . async—searchl [01 primary shard is not active Timeout: [1m Elasticsearch	5	543	February 16, 2023
Fatal Error in thread Elasticsearch Elasticsearch	8	2134	February 7, 2020
Elasticsearch crashing Elasticsearch	2	743	April 20, 2019

Elasticsearch stopped working

Related topics