# Random node failures **URL:** \ **Category:** Elasticsearch\ **Created:** [August 15, 2015, 5:00am UTC](https://discuss.elastic.co/t/random-node-failures/27439 "2015-08-15T05:00:37Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![R0B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/r0b/32/4203_2.png) [@R0B](https://discuss.elastic.co/u/R0B)\ **Post date:** [August 15, 2015, 5:00am UTC](https://discuss.elastic.co/t/random-node-failures/27439/1 "2015-08-15T05:00:37Z")
Recently brought up a 2 node cluster running 1.5.2-1 and things were working fine. Its been about 2 weeks now and for the past few days I have had to restart the nodes for the cluster to recover and become useable again. I am running 2 r3.xlarge instances within AWS each configured with a 1TB GP2 EBS volume and using the ec2 discovery method. When things start to go bad the last thing reported in the log is Caused by: org.elasticsearch.transport.TransportException: TransportService is closed stopped can't send request at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:270) When I look at the bigdesk plugin cluster health shows green however all I get from Kibana is a Gateway error when trying a search. If I restart elasticsearch things seem to be fine for around 20 hours or so and then the same situation reoccurs. I also have a single node cluster that I used as a test machine and it has never displayed this behavior. Can I provide further details/configuration to help diagnose if I have misconfigured in some way. Or help determine if I am hitting some kind of resource issue? -- Robert ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [August 15, 2015, 7:00am UTC](https://discuss.elastic.co/t/random-node-failures/27439/2 "2015-08-15T07:00:31Z")
What do your logs show? There should be more than that. ---
**Author:** ![R0B](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/r0b/32/4203_2.png) [@R0B](https://discuss.elastic.co/u/R0B)\ **Post date:** [August 15, 2015, 7:40am UTC](https://discuss.elastic.co/t/random-node-failures/27439/3 "2015-08-15T07:40:17Z")
Prior to the restart of the node there are pages and pages of below: [2015-08-14 20:31:10,323][DEBUG][action.admin.indices.stats] [esnode1] [logstash-2015.07.31][2], node[AZ1B\_kkRPqJJBI09XZ3GcM], [R], s[STARTED]: failed to execute [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@3bda42c5] org.elasticsearch.transport.SendRequestTransportException: [esnode2][inet[/esnode2:9300]][indices:monitor/stats[s]] at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:286) at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:249) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.performOperation(TransportBroadcastOperationAction.java:183) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction$AsyncBroadcastAction.start(TransportBroadcastOperationAction.java:151) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction.doExecute(TransportBroadcastOperationAction.java:71) at org.elasticsearch.action.support.broadcast.TransportBroadcastOperationAction.doExecute(TransportBroadcastOperationAction.java:47) at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:75) at org.elasticsearch.cluster.InternalClusterInfoService.updateIndicesStats(InternalClusterInfoService.java:267) at org.elasticsearch.cluster.InternalClusterInfoService$ClusterInfoUpdateJob.run(InternalClusterInfoService.java:356) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: org.elasticsearch.transport.TransportException: TransportService is closed stopped can't send request at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:270) ... 11 more But prior to above there is also this [2015-08-14 20:31:08,881][WARN][netty.channel.DefaultChannelPipeline] An exception was thrown by an exception handler. java.util.concurrent.RejectedExecutionException: Worker has already been shutdown at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.registerTask(AbstractNioSelector.java:120) at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:72) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36) at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:56) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36) at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioChannelSink.execute(AbstractNioChannelSink.java:34) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.execute(DefaultChannelPipeline.java:636) at org.elasticsearch.common.netty.channel.Channels.fireExceptionCaughtLater(Channels.java:496) at org.elasticsearch.common.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:46) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:658) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:781) at org.elasticsearch.common.netty.channel.Channels.write(Channels.java:725) at org.elasticsearch.common.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:71) at org.elasticsearch.common.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784) at org.elasticsearch.http.netty.pipelining.HttpPipeliningHandler.handleDownstream(HttpPipeliningHandler.java:87) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582) at org.elasticsearch.http.netty.NettyHttpChannel.sendResponse(NettyHttpChannel.java:199) at org.elasticsearch.rest.action.support.RestResponseListener.processResponse(RestResponseListener.java:43) at org.elasticsearch.rest.action.support.RestActionListener.onResponse(RestActionListener.java:49) ---
**Author:** ![dfr0](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dfr0/32/6433_2.png) [@dfr0](https://discuss.elastic.co/u/dfr0)\ **Post date:** [November 21, 2015, 1:10am UTC](https://discuss.elastic.co/t/random-node-failures/27439/4 "2015-11-21T01:10:25Z")
To me happen the same thing, i've the elasticsearch 2.0 version [2015-11-21 01:08:45,418][WARN][netty.channel.DefaultChannelPipeline] An exception was thrown by an exception handler. java.util.concurrent.RejectedExecutionException: Worker has already been shutdown at org.jboss.netty.channel.socket.nio.AbstractNioSelector.registerTask(AbstractNioSelector.java:120) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:72) at org.jboss.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36) at org.jboss.netty.channel.socket.nio.AbstractNioWorker.executeInIoThread(AbstractNioWorker.java:56) at org.jboss.netty.channel.socket.nio.NioWorker.executeInIoThread(NioWorker.java:36) at org.jboss.netty.channel.socket.nio.AbstractNioChannelSink.execute(AbstractNioChannelSink.java:34) at org.jboss.netty.channel.DefaultChannelPipeline.execute(DefaultChannelPipeline.java:636) at org.jboss.netty.channel.Channels.fireExceptionCaughtLater(Channels.java:496) at org.jboss.netty.channel.AbstractChannelSink.exceptionCaught(AbstractChannelSink.java:46) at org.jboss.netty.channel.DefaultChannelPipeline.notifyHandlerException(DefaultChannelPipeline.java:658) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:781) at org.jboss.netty.channel.Channels.write(Channels.java:725) at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:71) at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) at org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:784) at org.elasticsearch.http.netty.pipelining.HttpPipeliningHandler.handleDownstream(HttpPipeliningHandler.java:87) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591) at org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582) at org.elasticsearch.http.netty.NettyHttpChannel.sendResponse(NettyHttpChannel.java:174) at org.elasticsearch.rest.action.support.RestResponseListener.processResponse(RestResponseListener.java:43) at org.elasticsearch.rest.action.support.RestActionListener.onResponse(RestActionListener.java:49) at org.elasticsearch.action.bulk.TransportBulkAction$2.finishHim(TransportBulkAction.java:353) at org.elasticsearch.action.bulk.TransportBulkAction$2.onResponse(TransportBulkAction.java:325) at org.elasticsearch.action.bulk.TransportBulkAction$2.onResponse(TransportBulkAction.java:314) at org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicationPhase.doFinish(TransportReplicationAction.java:983) at org.elasticsearch.action.support.replication.TransportReplicationAction$ReplicationPhase.doRun(TransportReplicationAction.java:836) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.finishAndMoveToReplication(TransportReplicationAction.java:530) at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase.performOnPrimary(TransportReplicationAction.java:608) at org.elasticsearch.action.support.replication.TransportReplicationAction$PrimaryPhase$1.doRun(TransportReplicationAction.java:452) at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) ---
**Author:** ![dfr0](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dfr0/32/6433_2.png) [@dfr0](https://discuss.elastic.co/u/dfr0)\ **Post date:** [November 21, 2015, 6:01am UTC](https://discuss.elastic.co/t/random-node-failures/27439/5 "2015-11-21T06:01:47Z")
Resolved, i was a problem with the elasticsearch mapping and the reverse proxy. ---
**Author:** ![GSL10019](https://avatars.discourse-cdn.com/v4/letter/g/ecb155/32.png) [@GSL10019](https://discuss.elastic.co/u/GSL10019)\ **Post date:** [April 18, 2016, 6:51pm UTC](https://discuss.elastic.co/t/random-node-failures/27439/6 "2016-04-18T18:51:25Z")
Hi, how did u fix this particular problem by reverse porxy and mapping it? was it was quick fix in the yml for es? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 10:58pm UTC](https://discuss.elastic.co/t/random-node-failures/27439/7 "2017-07-05T22:58:24Z")