Node stopped and kicked out of cluster

Elastic Stack Elasticsearch
1

In a cluster of 3 nodes 1 node was automatically stopped and kicked out of the cluster. When I checked the log file this is what the exception was -

    [2017-02-23T04:15:57,244][ERROR][i.n.u.c.D.rejectedExecution] Failed to submit a listener notification task. Event loop shut down?
java.util.concurrent.RejectedExecutionException: event executor terminated
        at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:836) ~[netty-common-4.1.6.Final.jar:4.1.6.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:342) ~[netty-common-4.1.6.Final.jar:4.1.6.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:335) ~[netty-common-4.1.6.Final.jar:4.1.6.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:761) ~[netty-common-4.1.6.Final.jar:4.1.6.Final]
        at io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:767) [netty-common-4.1.6.Final.jar:4.1.6.Final]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:435) [netty-common-4.1.6.Final.jar:4.1.6.Final]
        at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:170) [netty-common-4.1.6.Final.jar:4.1.6.Final]
        at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:93) [netty-transport-4.1.6.Final.jar:4.1.6.Final]
        at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:28) [netty-transport-4.1.6.Final.jar:4.1.6.Final]
        at org.elasticsearch.transport.netty4.Netty4Transport.sendMessage(Netty4Transport.java:450) [transport-netty4-5.1.1.jar:5.1.1]
        at org.elasticsearch.transport.netty4.Netty4Transport.sendMessage(Netty4Transport.java:91) [transport-netty4-5.1.1.jar:5.1.1]
        at org.elasticsearch.transport.TcpTransport.internalSendMessage(TcpTransport.java:976) [elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.transport.TcpTransport.sendResponse(TcpTransport.java:1047) [elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.transport.TcpTransportChannel.sendResponse(TcpTransportChannel.java:67) [elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.transport.TcpTransportChannel.sendResponse(TcpTransportChannel.java:61) [elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.transport.DelegatingTransportChannel.sendResponse(DelegatingTransportChannel.java:58) [elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.transport.RequestHandlerRegistry$TransportChannelWrapper.sendResponse(RequestHandlerRegistry.java:111) [elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:301) [elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:297) [elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) [elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1385) [elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:527) [elasticsearch-5.1.1.jar:5.1.1]
        at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.1.1.jar:5.1.1]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_102]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_102]
        at java.lang.Thread.run(Thread.java:745) [?:1.8.0_102]
[2017-02-23T04:15:58,231][INFO ][o.e.n.Node               ] [nsbQKYA] stopped
[2017-02-23T04:15:58,231][INFO ][o.e.n.Node               ] [nsbQKYA] closing ...
[2017-02-23T04:15:58,255][INFO ][o.e.n.Node               ] [nsbQKYA] closed

I had to restart the Elasticsearch service to bring back the node into the cluster.
ES version - 5.1.1
OS - Linux 3.16.0-4-amd64
mlockall - true
index size - 6.7GB
RAM - 13GB(though top shows 12.5 is being used by ES which I am assuming is because of ES greedy caching).

Any ideas what could be happening and how to fix it?

2

Is there anything in the log before that?

3

The logs just indicate that the node was shutdown. Elasticsearch does not shut down by itself.

1 Like
4

This is the complete log file is at https://gist.github.com/Akarshit/eab3e9fd040944589afcd15a37058f73
I am also surprised why this happened, that's why I want to fix this.

5

Something is issuing a shutdown command to ES, the error above has nothing to do with it.
Try checking your OS level logs.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.