StackOverflowError in ES

Srinath_C · July 10, 2015, 4:10am

On our cluster of ES running 1.3.0 over 6 nodes, we observed a stackoverflow error on one of the nodes. The system has been under moderate bulk indexing operations.

The clients were receiving a Node disconnected exception for this node of ES and further drilling down into the logs showed the exception as shown below.

Had to restart the elasticsearch service but the cluster failed to recover to a green state. Atleast 2 indices were stuck initializing or relocating.

[2015-07-09 08:59:54,214][WARN ][org.elasticsearch        ] Exception cause unwrapping ran for 10 levels...
org.elasticsearch.transport.RemoteTransportException: [metrics-datastore-6-QA2906-perf][inet[/172.31.44.76:9300]][bulk/shard]
Caused by: org.elasticsearch.transport.RemoteTransportException: [metrics-datastore-3-QA2906-perf][inet[/172.31.34.173:9300]][bulk/shard]
Caused by: org.elasticsearch.transport.RemoteTransportException: [metrics-datastore-6-QA2906-perf][inet[/172.31.44.76:9300]][bulk/shard]
Caused by: org.elasticsearch.transport.RemoteTransportException: [metrics-datastore-3-QA2906-perf][inet[/172.31.34.173:9300]][bulk/shard]
Caused by: org.elasticsearch.transport.RemoteTransportException: [metrics-datastore-6-QA2906-perf][inet[/172.31.44.76:9300]][bulk/shard]
Caused by: org.elasticsearch.transport.RemoteTransportException: [metrics-datastore-3-QA2906-perf][inet[/172.31.34.173:9300]][bulk/shard]
Caused by: org.elasticsearch.transport.RemoteTransportException: [metrics-datastore-6-QA2906-perf][inet[/172.31.44.76:9300]][bulk/shard]
Caused by: org.elasticsearch.transport.RemoteTransportException: [metrics-datastore-3-QA2906-perf][inet[/172.31.34.173:9300]][bulk/shard]
Caused by: org.elasticsearch.transport.RemoteTransportException: [metrics-datastore-6-QA2906-perf][inet[/172.31.44.76:9300]][bulk/shard]
Caused by: org.elasticsearch.transport.RemoteTransportException: [metrics-datastore-3-QA2906-perf][inet[/172.31.34.173:9300]][bulk/shard]
Caused by: org.elasticsearch.transport.RemoteTransportException: [metrics-datastore-6-QA2906-perf][inet[/172.31.44.76:9300]][bulk/shard]
Caused by: org.elasticsearch.transport.RemoteTransportException: [metrics-datastore-3-QA2906-perf][inet[/172.31.34.173:9300]][bulk/shard]
Caused by: org.elasticsearch.transport.RemoteTransportException: [metrics-datastore-3-QA2906-perf][inet[/172.31.34.173:9300]][bulk/shard]
Caused by: org.elasticsearch.transport.RemoteTransportException: [metrics-datastore-6-QA2906-perf][inet[/172.31.44.76:9300]][bulk/shard]
Caused by: org.elasticsearch.transport.RemoteTransportException: [metrics-datastore-3-QA2906-perf][inet[/172.31.34.173:9300]][bulk/shard]
Caused by: org.elasticsearch.transport.RemoteTransportException: [metrics-datastore-6-QA2906-perf][inet[/172.31.44.76:9300]][bulk/shard]
Caused by: org.elasticsearch.transport.RemoteTransportException: [metrics-datastore-3-QA2906-perf][inet[/172.31.34.173:9300]][bulk/shard]
Caused by: org.elasticsearch.transport.RemoteTransportException: [metrics-datastore-6-QA2906-perf][inet[/172.31.44.76:9300]][bulk/shard]
Caused by: org.elasticsearch.transport.RemoteTransportException: [metrics-datastore-3-QA2906-perf][inet[/172.31.34.173:9300]][bulk/shard]
Caused by: org.elasticsearch.transport.RemoteTransportException: [metrics-datastore-6-QA2906-perf][inet[/172.31.44.76:9300]][bulk/shard]
Caused by: org.elasticsearch.transport.RemoteTransportException: [metrics-datastore-3-QA2906-perf][inet[/172.31.34.173:9300]][bulk/shard]
Caused by: org.elasticsearch.transport.RemoteTransportException: [metrics-datastore-6-QA2906-perf][inet[/172.31.44.76:9300]][bulk/shard]
Caused by: org.elasticsearch.transport.RemoteTransportException: [metrics-datastore-3-QA2906-perf][inet[/172.31.34.173:9300]][bulk/shard]
Caused by: org.elasticsearch.transport.RemoteTransportException: [metrics-datastore-6-QA2906-perf][inet[/172.31.44.76:9300]][bulk/shard]
Caused by: org.elasticsearch.transport.RemoteTransportException: [metrics-datastore-3-QA2906-perf][inet[/172.31.34.173:9300]][bulk/shard]
Caused by: org.elasticsearch.transport.RemoteTransportException: [metrics-datastore-6-QA2906-perf][inet[/172.31.44.76:9300]][bulk/shard]
Caused by: org.elasticsearch.transport.RemoteTransportException: [metrics-datastore-3-QA2906-perf][inet[/172.31.34.173:9300]][bulk/shard]
Caused by: org.elasticsearch.transport.RemoteTransportException: [metrics-datastore-6-QA2906-perf][inet[/172.31.44.76:9300]][bulk/shard]
Caused by: org.elasticsearch.transport.RemoteTransportException: [metrics-datastore-3-QA2906-perf][inet[/172.31.34.173:9300]][bulk/shard]
Caused by: org.elasticsearch.transport.RemoteTransportException: [metrics-datastore-6-QA2906-perf][inet[/172.31.44.76:9300]][bulk/shard]
Caused by: org.elasticsearch.transport.RemoteTransportException: Failed to deserialize exception response from stream
Caused by: org.elasticsearch.transport.TransportSerializationException: Failed to deserialize exception response from stream
        at org.elasticsearch.transport.netty.MessageChannelHandler.handlerResponseError(MessageChannelHandler.java:173)
        at org.elasticsearch.transport.netty.MessageChannelHandler.messageReceived(MessageChannelHandler.java:125)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
        at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:462)
        at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:443)
        at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:303)
        at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
        at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
        at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
        at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
        at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.StackOverflowError

warkolm · July 10, 2015, 4:21am

It mentions serialisation problems, which can be caused by different JVM or client versions, so it might be best to check those.

Srinath_C · July 10, 2015, 8:25am

JVM and client version mismatches can be ruled out. Both server and client are of 1.3.0. The systems have been running for quite some time and only after a constant load this exception showed up.

Could this be an issue with ES? What is it serializing/deserializing?

Jason_Wee · July 12, 2015, 1:19pm

had similar exception as yours , read here https://github.com/elastic/elasticsearch/issues/4639 though this does not happen anymore, not even sure how to reproduce. hope it will give additional information to pin down the source of problem.

Srinath_C · July 12, 2015, 5:47pm

Thanks @Jason_Wee. Will update that ticket to provide more info.

Topic		Replies	Views
Fatal error in thread java.lang.StackOverflowError:null Elasticsearch	1	1096	January 14, 2019
Failed Shard Recovery Elasticsearch	5	3181	July 6, 2017
Get a java.lang.StackOverflowError when indexing in ES Elasticsearch	5	1480	July 5, 2017
Exception cause unwrapping ran for 10 levels Elasticsearch	7	753	July 6, 2017
ES 5.1 - Shard recovery stuck in INIT- IndexShardRelocatedException: Already relocated Elasticsearch	2	826	November 3, 2017

StackOverflowError in ES

Related Topics