# Node stopped and kicked out of cluster **URL:** \ **Category:** Elasticsearch\ **Created:** [February 23, 2017, 11:34am UTC](https://discuss.elastic.co/t/node-stopped-and-kicked-out-of-cluster/76205 "2017-02-23T11:34:14Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Akarshit\_Wal](https://avatars.discourse-cdn.com/v4/letter/a/ecae2f/32.png) [@Akarshit\_Wal](https://discuss.elastic.co/u/Akarshit_Wal)\ **Post date:** [February 23, 2017, 11:34am UTC](https://discuss.elastic.co/t/node-stopped-and-kicked-out-of-cluster/76205/1 "2017-02-23T11:34:14Z")
In a cluster of 3 nodes 1 node was automatically stopped and kicked out of the cluster. When I checked the log file this is what the exception was - ```auto [2017-02-23T04:15:57,244][ERROR][i.n.u.c.D.rejectedExecution] Failed to submit a listener notification task. Event loop shut down? java.util.concurrent.RejectedExecutionException: event executor terminated at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:836) ~[netty-common-4.1.6.Final.jar:4.1.6.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:342) ~[netty-common-4.1.6.Final.jar:4.1.6.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:335) ~[netty-common-4.1.6.Final.jar:4.1.6.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:761) ~[netty-common-4.1.6.Final.jar:4.1.6.Final] at io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:767) [netty-common-4.1.6.Final.jar:4.1.6.Final] at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:435) [netty-common-4.1.6.Final.jar:4.1.6.Final] at io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:170) [netty-common-4.1.6.Final.jar:4.1.6.Final] at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:93) [netty-transport-4.1.6.Final.jar:4.1.6.Final] at io.netty.channel.DefaultChannelPromise.addListener(DefaultChannelPromise.java:28) [netty-transport-4.1.6.Final.jar:4.1.6.Final] at org.elasticsearch.transport.netty4.Netty4Transport.sendMessage(Netty4Transport.java:450) [transport-netty4-5.1.1.jar:5.1.1] at org.elasticsearch.transport.netty4.Netty4Transport.sendMessage(Netty4Transport.java:91) [transport-netty4-5.1.1.jar:5.1.1] at org.elasticsearch.transport.TcpTransport.internalSendMessage(TcpTransport.java:976) [elasticsearch-5.1.1.jar:5.1.1] at org.elasticsearch.transport.TcpTransport.sendResponse(TcpTransport.java:1047) [elasticsearch-5.1.1.jar:5.1.1] at org.elasticsearch.transport.TcpTransportChannel.sendResponse(TcpTransportChannel.java:67) [elasticsearch-5.1.1.jar:5.1.1] at org.elasticsearch.transport.TcpTransportChannel.sendResponse(TcpTransportChannel.java:61) [elasticsearch-5.1.1.jar:5.1.1] at org.elasticsearch.transport.DelegatingTransportChannel.sendResponse(DelegatingTransportChannel.java:58) [elasticsearch-5.1.1.jar:5.1.1] at org.elasticsearch.transport.RequestHandlerRegistry$TransportChannelWrapper.sendResponse(RequestHandlerRegistry.java:111) [elasticsearch-5.1.1.jar:5.1.1] at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:301) [elasticsearch-5.1.1.jar:5.1.1] at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:297) [elasticsearch-5.1.1.jar:5.1.1] at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:69) [elasticsearch-5.1.1.jar:5.1.1] at org.elasticsearch.transport.TcpTransport$RequestHandler.doRun(TcpTransport.java:1385) [elasticsearch-5.1.1.jar:5.1.1] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:527) [elasticsearch-5.1.1.jar:5.1.1] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-5.1.1.jar:5.1.1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_102] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_102] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_102] [2017-02-23T04:15:58,231][INFO][o.e.n.Node] [nsbQKYA] stopped [2017-02-23T04:15:58,231][INFO][o.e.n.Node] [nsbQKYA] closing ... [2017-02-23T04:15:58,255][INFO][o.e.n.Node] [nsbQKYA] closed ``` I had to restart the Elasticsearch service to bring back the node into the cluster. ES version - 5.1.1 OS - Linux 3.16.0-4-amd64 mlockall - true index size - 6.7GB RAM - 13GB(though top shows 12.5 is being used by ES which I am assuming is because of ES greedy caching). Any ideas what could be happening and how to fix it? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [February 24, 2017, 10:05am UTC](https://discuss.elastic.co/t/node-stopped-and-kicked-out-of-cluster/76205/2 "2017-02-24T10:05:43Z")
Is there anything in the log before that? ---
**Author:** ![ywelsch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ywelsch/32/7751_2.png) [@ywelsch](https://discuss.elastic.co/u/ywelsch)\ **Post date:** [February 24, 2017, 10:14am UTC](https://discuss.elastic.co/t/node-stopped-and-kicked-out-of-cluster/76205/3 "2017-02-24T10:14:46Z")
The logs just indicate that the node was shutdown. Elasticsearch does not shut down by itself. ---
**Author:** ![Akarshit\_Wal](https://avatars.discourse-cdn.com/v4/letter/a/ecae2f/32.png) [@Akarshit\_Wal](https://discuss.elastic.co/u/Akarshit_Wal)\ **Post date:** [February 24, 2017, 11:43am UTC](https://discuss.elastic.co/t/node-stopped-and-kicked-out-of-cluster/76205/4 "2017-02-24T11:43:31Z")
This is the complete log file is at [https://gist.github.com/Akarshit/eab3e9fd040944589afcd15a37058f73](https://gist.github.com/Akarshit/eab3e9fd040944589afcd15a37058f73) I am also surprised why this happened, that's why I want to fix this. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [February 24, 2017, 9:49pm UTC](https://discuss.elastic.co/t/node-stopped-and-kicked-out-of-cluster/76205/5 "2017-02-24T21:49:05Z")
Something is issuing a shutdown command to ES, the error above has nothing to do with it. Try checking your OS level logs. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 24, 2017, 9:49pm UTC](https://discuss.elastic.co/t/node-stopped-and-kicked-out-of-cluster/76205/6 "2017-03-24T21:49:17Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.