# "received shard failed for... \[RemoteTransportException\]" for the node on the same machine **URL:** \ **Category:** Elasticsearch\ **Created:** [August 25, 2013, 8:52am UTC](https://discuss.elastic.co/t/received-shard-failed-for-remotetransportexception-for-the-node-on-the-same-machine/13323 "2013-08-25T08:52:20Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![slimsuperhero](https://avatars.discourse-cdn.com/v4/letter/s/77aa72/32.png) [@slimsuperhero](https://discuss.elastic.co/u/slimsuperhero)\ **Post date:** [August 25, 2013, 8:52am UTC](https://discuss.elastic.co/t/received-shard-failed-for-remotetransportexception-for-the-node-on-the-same-machine/13323/1 "2013-08-25T08:52:20Z")
Hello! I have develop setup of elasticsearch with every node running on the single server. It has elasticsearch master instance, which balances the load between two elasticsearch data instances (6 shards and 1 replica per index). Sometimes I see the following error in the log: [2013-08-25 12:30:29,549][WARN][cluster.action.shard] [es\_master] received shard failed for [logstash-2013.08.25][4], node[x2u8Oq70T4iELaMs\_9FFUA], [R], s[STARTED], reason [Failed to perform [index] on replica, message [RemoteTransportException[[es\_node2][inet[/192.168.2.221:9301]][index/replica]]; nested: EsRejectedExecutionException[rejected execution of [org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler]]; ]] It is always the same instance (es\_node2) and failed index on replica. RemoteTransportException seems very strange to me, since es\_node2 is on the same machine with the master and es\_node1. And I have never seen such error for es\_node1 (settings for es\_node1 and es\_node2 are the same). I'm indexing about 3,5K events per second, so I have socket buffers memory increased. Please explain me what does "EsRejectedExecutionException[rejected execution of [org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler]]" mean? Any help would be highly appreciated. Thanks! -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![R\_Toma](https://avatars.discourse-cdn.com/v4/letter/r/e480ec/32.png) [@R\_Toma](https://discuss.elastic.co/u/R_Toma)\ **Post date:** [September 17, 2013, 12:33pm UTC](https://discuss.elastic.co/t/received-shard-failed-for-remotetransportexception-for-the-node-on-the-same-machine/13323/2 "2013-09-17T12:33:33Z")
Hi Gennady, Did you get anywhere with this issue? I have the same issue. I have tried setting threadpool.index.queue\_size=-1 on ES 0.20.x as described on [http://jontai.me/blog/2013/06/esrejectedexecutionexception-rejected-execution-of-messagechannelhandler-requesthandler/](http://jontai.me/blog/2013/06/esrejectedexecutionexception-rejected-execution-of-messagechannelhandler-requesthandler/) but no luck so far. Regards, Renzo Op zondag 25 augustus 2013 10:52:20 UTC+2 schreef Gennady Alexandrov: > Hello! > > I have develop setup of elasticsearch with every node running on the > single server. > It has elasticsearch master instance, which balances the load between two > elasticsearch data instances (6 shards and 1 replica per index). > Sometimes I see the following error in the log: > > [2013-08-25 12:30:29,549][WARN][cluster.action.shard] [es\_master] > received shard failed for [logstash-2013.08.25][4], > node[x2u8Oq70T4iELaMs\_9FFUA], [R], s[STARTED], reason [Failed to perform > [index] on replica, message > [RemoteTransportException[[es\_node2][inet[/192.168.2.221:9301]][index/replica]]; > nested: EsRejectedExecutionException[rejected execution of > [org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler]]; > ]] > > It is always the same instance (es\_node2) and failed index on replica. > RemoteTransportException seems very strange to me, since es\_node2 is on > the same machine with the master and es\_node1. And I have never seen such > error for es\_node1 (settings for es\_node1 and es\_node2 are the same). I'm > indexing about 3,5K events per second, so I have socket buffers memory > increased. > > Please explain me what does "EsRejectedExecutionException[rejected > execution of > [org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler]]" > mean? > > Any help would be highly appreciated. > > Thanks! -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![slimsuperhero](https://avatars.discourse-cdn.com/v4/letter/s/77aa72/32.png) [@slimsuperhero](https://discuss.elastic.co/u/slimsuperhero)\ **Post date:** [September 17, 2013, 12:40pm UTC](https://discuss.elastic.co/t/received-shard-failed-for-remotetransportexception-for-the-node-on-the-same-machine/13323/3 "2013-09-17T12:40:05Z")
Hi! Yes, I've managed with this. Setting threadpool.index.queue\_size=-1 wouldn't fix it in 0.20.6. I have to set large value explicitly. In my case threadpool.index.queue\_size: 150000 is fine. 2013/9/17 R. Toma [renzo.toma@gmail.com](mailto:renzo.toma@gmail.com) > Hi Gennady, > > Did you get anywhere with this issue? I have the same issue. > > I have tried setting threadpool.index.queue\_size=-1 on ES 0.20.x as > described on > [http://jontai.me/blog/2013/06/esrejectedexecutionexception-rejected-execution-of-messagechannelhandler-requesthandler/but](http://jontai.me/blog/2013/06/esrejectedexecutionexception-rejected-execution-of-messagechannelhandler-requesthandler/but) no luck so far. > > Regards, > Renzo > > Op zondag 25 augustus 2013 10:52:20 UTC+2 schreef Gennady Alexandrov: > > > Hello! > > > > I have develop setup of elasticsearch with every node running on the > > single server. > > It has elasticsearch master instance, which balances the load between two > > elasticsearch data instances (6 shards and 1 replica per index). > > Sometimes I see the following error in the log: > > > > [2013-08-25 12:30:29,549][WARN][cluster.action.shard] [es\_master] > > received shard failed for [logstash-2013.08.25][4], > > node[x2u8Oq70T4iELaMs\_9FFUA], [R], s[STARTED], reason [Failed to perform > > [index] on replica, message [RemoteTransportException[[es\_\*\*node2][inet[/ > > 192.168.2.221:\*\*9301]][index/replica]]; nested: > > EsRejectedExecutionException[\*\*rejected execution of > > [org.elasticsearch.transport.\*\*netty.MessageChannelHandler$\*\*RequestHandler]]; > > ]] > > > > It is always the same instance (es\_node2) and failed index on replica. > > RemoteTransportException seems very strange to me, since es\_node2 is on > > the same machine with the master and es\_node1. And I have never seen such > > error for es\_node1 (settings for es\_node1 and es\_node2 are the same). I'm > > indexing about 3,5K events per second, so I have socket buffers memory > > increased. > > > > Please explain me what does "EsRejectedExecutionException[\*\*rejected > > execution of [org.elasticsearch.transport.\*_netty.MessageChannelHandler$_ > > \*RequestHandler]]" mean? > > > > Any help would be highly appreciated. > > > > Thanks! > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/9lyPyvpeTSY/unsubscribe](https://groups.google.com/d/topic/elasticsearch/9lyPyvpeTSY/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- _Best regards,_\* \* _Gennady Aleksandrov_ -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![R\_Toma](https://avatars.discourse-cdn.com/v4/letter/r/e480ec/32.png) [@R\_Toma](https://discuss.elastic.co/u/R_Toma)\ **Post date:** [September 17, 2013, 1:03pm UTC](https://discuss.elastic.co/t/received-shard-failed-for-remotetransportexception-for-the-node-on-the-same-machine/13323/4 "2013-09-17T13:03:07Z")
Hi, Aaaah, that explains it 🙂 Will try with large value. Thank you! Regards, Renzo Op dinsdag 17 september 2013 14:40:05 UTC+2 schreef Gennady Alexandrov: > Hi! > > Yes, I've managed with this. > Setting threadpool.index.queue\_size=-1 wouldn't fix it in 0.20.6. I have > to set large value explicitly. In my case threadpool.index.queue\_size: > 150000 is fine. > > 2013/9/17 R. Toma \<[renzo...@gmail.com](mailto:renzo...@gmail.com) \\> > > > Hi Gennady, > > > > Did you get anywhere with this issue? I have the same issue. > > > > I have tried setting threadpool.index.queue\_size=-1 on ES 0.20.x as > > described on > > [http://jontai.me/blog/2013/06/esrejectedexecutionexception-rejected-execution-of-messagechannelhandler-requesthandler/but](http://jontai.me/blog/2013/06/esrejectedexecutionexception-rejected-execution-of-messagechannelhandler-requesthandler/but) no luck so far. > > > > Regards, > > Renzo > > > > Op zondag 25 augustus 2013 10:52:20 UTC+2 schreef Gennady Alexandrov: > > > > > Hello! > > > > > > I have develop setup of elasticsearch with every node running on the > > > single server. > > > It has elasticsearch master instance, which balances the load between > > > two elasticsearch data instances (6 shards and 1 replica per index). > > > Sometimes I see the following error in the log: > > > > > > [2013-08-25 12:30:29,549][WARN][cluster.action.shard] [es\_master] > > > received shard failed for [logstash-2013.08.25][4], > > > node[x2u8Oq70T4iELaMs\_9FFUA], [R], s[STARTED], reason [Failed to perform > > > [index] on replica, message [RemoteTransportException[[es\_\*\* > > > node2][inet[/192.168.2.221:\*\*9301]][index/replica]]; nested: > > > EsRejectedExecutionException[\*\*rejected execution of > > > [org.elasticsearch.transport.\*\*netty.MessageChannelHandler$\*\*RequestHandler]]; > > > ]] > > > > > > It is always the same instance (es\_node2) and failed index on replica. > > > RemoteTransportException seems very strange to me, since es\_node2 is on > > > the same machine with the master and es\_node1. And I have never seen such > > > error for es\_node1 (settings for es\_node1 and es\_node2 are the same). I'm > > > indexing about 3,5K events per second, so I have socket buffers memory > > > increased. > > > > > > Please explain me what does "EsRejectedExecutionException[\*\*rejected > > > execution of [org.elasticsearch.transport.\*\*netty.MessageChannelHandler$ > > > \*\*RequestHandler]]" mean? > > > > > > Any help would be highly appreciated. > > > > > > Thanks! > > > > -- > > You received this message because you are subscribed to a topic in the > > Google Groups "elasticsearch" group. > > To unsubscribe from this topic, visit > > [https://groups.google.com/d/topic/elasticsearch/9lyPyvpeTSY/unsubscribe](https://groups.google.com/d/topic/elasticsearch/9lyPyvpeTSY/unsubscribe). > > To unsubscribe from this group and all its topics, send an email to > > [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > _Best regards,_\* > \* > _Gennady Aleksandrov_ -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![slimsuperhero](https://avatars.discourse-cdn.com/v4/letter/s/77aa72/32.png) [@slimsuperhero](https://discuss.elastic.co/u/slimsuperhero)\ **Post date:** [September 17, 2013, 1:03pm UTC](https://discuss.elastic.co/t/received-shard-failed-for-remotetransportexception-for-the-node-on-the-same-machine/13323/5 "2013-09-17T13:03:53Z")
Good luck! 2013/9/17 R. Toma [renzo.toma@gmail.com](mailto:renzo.toma@gmail.com) > Hi, > > Aaaah, that explains it 🙂 Will try with large value. > > Thank you! > > Regards, > Renzo > > Op dinsdag 17 september 2013 14:40:05 UTC+2 schreef Gennady Alexandrov: > > > Hi! > > > > Yes, I've managed with this. > > Setting threadpool.index.queue\_size=-1 wouldn't fix it in 0.20.6. I have > > to set large value explicitly. In my case threadpool.index.queue\_size: > > 150000 is fine. > > > > 2013/9/17 R. Toma [renzo...@gmail.com](mailto:renzo...@gmail.com) > > > > > Hi Gennady, > > > > > > Did you get anywhere with this issue? I have the same issue. > > > > > > I have tried setting threadpool.index.queue\_size=-1 on ES 0.20.x as > > > described on [http://jontai.me/blog/2013/06/](http://jontai.me/blog/2013/06/)\*\* > > > esrejectedexecutionexception- **rejected-execution-of-** > > > messagechannelhandler-\*\*requesthandler/[http://jontai.me/blog/2013/06/esrejectedexecutionexception-rejected-execution-of-messagechannelhandler-requesthandler/](http://jontai.me/blog/2013/06/esrejectedexecutionexception-rejected-execution-of-messagechannelhandler-requesthandler/)but no luck so far. > > > > > > Regards, > > > Renzo > > > > > > Op zondag 25 augustus 2013 10:52:20 UTC+2 schreef Gennady Alexandrov: > > > > > > > Hello! > > > > > > > > I have develop setup of elasticsearch with every node running on the > > > > single server. > > > > It has elasticsearch master instance, which balances the load between > > > > two elasticsearch data instances (6 shards and 1 replica per index). > > > > Sometimes I see the following error in the log: > > > > > > > > [2013-08-25 12:30:29,549][WARN][cluster.action.shard] [es\_master] > > > > received shard failed for [logstash-2013.08.25][4], > > > > node[x2u8Oq70T4iELaMs\_9FFUA], [R], s[STARTED], reason [Failed to perform > > > > [index] on replica, message [RemoteTransportException[[es\_\*\*\*\* > > > > node2][inet[/192.168.2.221: **930** 1]][index/replica]]; nested: > > > > EsRejectedExecutionException[ **r** ejected execution of > > > > [org.elasticsearch.transport. **n** etty.MessageChannelHandler$ **Req** uestHandler]]; > > > > ]] > > > > > > > > It is always the same instance (es\_node2) and failed index on replica. > > > > RemoteTransportException seems very strange to me, since es\_node2 is on > > > > the same machine with the master and es\_node1. And I have never seen such > > > > error for es\_node1 (settings for es\_node1 and es\_node2 are the same). I'm > > > > indexing about 3,5K events per second, so I have socket buffers memory > > > > increased. > > > > > > > > Please explain me what does "EsRejectedExecutionException[\*\*\*\*rejected > > > > execution of [org.elasticsearch.transport. **n** > > > > etty.MessageChannelHandler$ **Req** uestHandler]]" mean? > > > > > > > > Any help would be highly appreciated. > > > > > > > > Thanks! > > > > > > -- > > > You received this message because you are subscribed to a topic in the > > > Google Groups "elasticsearch" group. > > > To unsubscribe from this topic, visit [https://groups.google.com/d/](https://groups.google.com/d/)\*\* > > > topic/elasticsearch/\*\*9lyPyvpeTSY/unsubscribe[https://groups.google.com/d/topic/elasticsearch/9lyPyvpeTSY/unsubscribe](https://groups.google.com/d/topic/elasticsearch/9lyPyvpeTSY/unsubscribe) > > > . > > > To unsubscribe from this group and all its topics, send an email to > > > elasticsearc...@\*\*[googlegroups.com](http://googlegroups.com). > > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > > . > > > > -- > > _Best regards,_\* > > \* > > _Gennady Aleksandrov_ > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > [https://groups.google.com/d/topic/elasticsearch/9lyPyvpeTSY/unsubscribe](https://groups.google.com/d/topic/elasticsearch/9lyPyvpeTSY/unsubscribe). > To unsubscribe from this group and all its topics, send an email to > [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- _Best regards,_\* \* _Gennady Aleksandrov_ -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:16am UTC](https://discuss.elastic.co/t/received-shard-failed-for-remotetransportexception-for-the-node-on-the-same-machine/13323/6 "2017-07-06T02:16:02Z")