Elasticsearch crashes with StackOverflowError

praveenmak · October 5, 2017, 6:59pm

Version Used: ElasticSearch 5.6.6

Mode: Single Node, Single Cluster - config is simple as shown below.

node.master: true
node.data: true
transport.type: local # disable network
bootstrap.memory_lock: true
thread_pool.search.queue_size: 10000

I am trying to search in Kibana with "esplogs-*", there are 75 Million records.

Here is the stacktrace after searching,

[2017-10-05T14:50:07,327][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [localhost] fatal error in thread [elasticsearch[localhost][search][T#19]], exiting
java.lang.StackOverflowError: null
at java.util.Collections$UnmodifiableCollection.size(Collections.java:1030) ~[?:1.8.0_144]
at java.util.Collections$UnmodifiableCollection.size(Collections.java:1030) ~[?:1.8.0_144]
at org.elasticsearch.cluster.routing.PlainShardsIterator.remaining(PlainShardsIterator.java:50) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.successfulShardExecution(InitialSearchPhase.java:196) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.skipShard(InitialSearchPhase.java:323) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.skipShard(AbstractSearchAsyncAction.java:321) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.maybeExecuteNext(InitialSearchPhase.java:147) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.successfulShardExecution(InitialSearchPhase.java:207) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.skipShard(InitialSearchPhase.java:323) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.skipShard(AbstractSearchAsyncAction.java:321) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.maybeExecuteNext(InitialSearchPhase.java:147) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.successfulShardExecution(InitialSearchPhase.java:207) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.skipShard(InitialSearchPhase.java:323) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.skipShard(AbstractSearchAsyncAction.java:321) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.maybeExecuteNext(InitialSearchPhase.java:147) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.successfulShardExecution(InitialSearchPhase.java:207) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.skipShard(InitialSearchPhase.java:323) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.skipShard(AbstractSearchAsyncAction.java:321) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.maybeExecuteNext(InitialSearchPhase.java:147) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.successfulShardExecution(InitialSearchPhase.java:207) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.skipShard(InitialSearchPhase.java:323) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.skipShard(AbstractSearchAsyncAction.java:321) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.maybeExecuteNext(InitialSearchPhase.java:147) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.successfulShardExecution(InitialSearchPhase.java:207) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.skipShard(InitialSearchPhase.java:323) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.skipShard(AbstractSearchAsyncAction.java:321) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.maybeExecuteNext(InitialSearchPhase.java:147) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.successfulShardExecution(InitialSearchPhase.java:207) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.skipShard(InitialSearchPhase.java:323) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.skipShard(AbstractSearchAsyncAction.java:321) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.maybeExecuteNext(InitialSearchPhase.java:147) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.successfulShardExecution(InitialSearchPhase.java:207) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.skipShard(InitialSearchPhase.java:323) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.skipShard(AbstractSearchAsyncAction.java:321) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.maybeExecuteNext(InitialSearchPhase.java:147) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.successfulShardExecution(InitialSearchPhase.java:207) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.skipShard(InitialSearchPhase.java:323) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.skipShard(AbstractSearchAsyncAction.java:321) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.maybeExecuteNext(InitialSearchPhase.java:147) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.successfulShardExecution(InitialSearchPhase.java:207) ~[elasticsearch-5.6.2.jar:5.6.2]
at org.elasticsearch.action.search.InitialSearchPhase.skipShard(InitialSearchPhase.java:323) ~[elasticsearch-5.6.2.jar:5.6.2]

Hoping if someone helps. Please let me know what you need from me.

praveenmak · October 5, 2017, 10:51pm

How do I check how many shards are occupied..

I keep getting this error whenever I insert loads of data. I have created 24 indexes and I have 5 shards (default).
Is there a better configuration to handle single node with large data?

Please let me know.

Rohithnama · October 5, 2017, 11:06pm

I had the same issue but I have an option to delete the indexes anytime so whenever I encountered this scenario, I deleted the indices- it is a temporary solution. If you don't mind deleting the some of the indexes just give a shot!

The path to delete is ~/elasticsearch-5.6.2/data/nodes/0/indices. I'm not sure if this could resolve your issues but it's worth the try!

warkolm · October 5, 2017, 11:25pm

Please check again, we are only at 5.6.2.

That is dangerous, why did you increase it?

Is there more errors at this time? What do the Kibana logs show?

Don't do that, it's very dangerous and can cause data loss. If you need to delete data, use the APIs.

praveenmak · October 6, 2017, 3:34pm

Thanks @Rohithnama and @warkolm for response.

You are right my bad, I typed it wrong its 5.6.2

/usr/share/elasticsearch/bin/elasticsearch --version
Version: 5.6.2, Build: 57e20f3/2017-09-23T13:16:45.703Z, JVM: 1.8.0_144

I saw a suggestion , increasing the thread_pool.search.queue_size fixed the slowness.
https://stackoverflow.com/questions/30053967/courier-fetch-shards-failed

warkolm · October 6, 2017, 8:36pm

No, it just overloads things even more.

praveenmak · October 13, 2017, 3:03pm

@warkolm

I am seeing another crash , I had reduced the queue size to 1000. Could you please help.

[2017-10-13T10:46:38,970][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [bctlppostalclean18] fatal error in thread [elasticsearch[bctlppostalclean18][search][T#10]], exiting
java.lang.StackOverflowError: null
at org.elasticsearch.cluster.routing.PlainShardsIterator.remaining(PlainShardsIterator.java:50) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.action.search.InitialSearchPhase.successfulShardExecution(InitialSearchPhase.java:196) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.action.search.InitialSearchPhase.skipShard(InitialSearchPhase.java:323) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.skipShard(AbstractSearchAsyncAction.java:321) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.action.search.InitialSearchPhase.maybeExecuteNext(InitialSearchPhase.java:147) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.action.search.InitialSearchPhase.successfulShardExecution(InitialSearchPhase.java:207) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.action.search.InitialSearchPhase.skipShard(InitialSearchPhase.java:323) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.skipShard(AbstractSearchAsyncAction.java:321) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.action.search.InitialSearchPhase.maybeExecuteNext(InitialSearchPhase.java:147) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.action.search.InitialSearchPhase.successfulShardExecution(InitialSearchPhase.java:207) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.action.search.InitialSearchPhase.skipShard(InitialSearchPhase.java:323) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.skipShard(AbstractSearchAsyncAction.java:321) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.action.search.InitialSearchPhase.maybeExecuteNext(InitialSearchPhase.java:147) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.action.search.InitialSearchPhase.successfulShardExecution(InitialSearchPhase.java:207) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.action.search.InitialSearchPhase.skipShard(InitialSearchPhase.java:323) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.skipShard(AbstractSearchAsyncAction.java:321) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.action.search.InitialSearchPhase.maybeExecuteNext(InitialSearchPhase.java:147) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.action.search.InitialSearchPhase.successfulShardExecution(InitialSearchPhase.java:207) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.action.search.InitialSearchPhase.skipShard(InitialSearchPhase.java:323) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.skipShard(AbstractSearchAsyncAction.java:321) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.action.search.InitialSearchPhase.maybeExecuteNext(InitialSearchPhase.java:147) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.action.search.InitialSearchPhase.successfulShardExecution(InitialSearchPhase.java:207) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.action.search.InitialSearchPhase.skipShard(InitialSearchPhase.java:323) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.skipShard(AbstractSearchAsyncAction.java:321) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.action.search.InitialSearchPhase.maybeExecuteNext(InitialSearchPhase.java:147) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.action.search.InitialSearchPhase.successfulShardExecution(InitialSearchPhase.java:207) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.action.search.InitialSearchPhase.skipShard(InitialSearchPhase.java:323) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.skipShard(AbstractSearchAsyncAction.java:321) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.action.search.InitialSearchPhase.maybeExecuteNext(InitialSearchPhase.java:147) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.action.search.InitialSearchPhase.successfulShardExecution(InitialSearchPhase.java:207) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.action.search.InitialSearchPhase.skipShard(InitialSearchPhase.java:323) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.skipShard(AbstractSearchAsyncAction.java:321) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.action.search.InitialSearchPhase.maybeExecuteNext(InitialSearchPhase.java:147) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.action.search.InitialSearchPhase.successfulShardExecution(InitialSearchPhase.java:207) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.action.search.InitialSearchPhase.skipShard(InitialSearchPhase.java:323) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.action.search.AbstractSearchAsyncAction.skipShard(AbstractSearchAsyncAction.java:321) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.action.search.InitialSearchPhase.maybeExecuteNext(InitialSearchPhase.java:147) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.action.search.InitialSearchPhase.successfulShardExecution(InitialSearchPhase.java:207) ~[elasticsearch-5.6.3.jar:5.6.3]
at org.elasticsearch.action.search.InitialSearchPhase.skipShard(InitialSearchPhase.java:323) ~[elasticsearch-5.6.3.jar:5.6.3]

praveenmak · October 17, 2017, 7:08pm

Please help , is this a known issue?

erley · October 24, 2017, 3:36pm

We have the same problem (ES v5.6.2).
A simple license install breaks ES nodes with StackOverflowError
Currently we had to rollback to v5.5.2 - the last known to work correctly...

warkolm · October 24, 2017, 6:15pm

Please create a new thread with some more detail, it may not be related to this one so separating it out would be super helpful in trying to find the problem

praveenmak · October 24, 2017, 8:01pm

@warkolm

May I know if this issue is a known issue or not?

warkolm · October 24, 2017, 8:26pm

I haven't seen it before, but that's not a definitive answer sorry. Did you look on github?

praveenmak · October 24, 2017, 8:30pm

Could not find this in Github

system · November 21, 2017, 8:30pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.