"received shard failed for... [RemoteTransportException]" for the node on the same machine

Elastic Stack Elasticsearch
1

Hello!

I have develop setup of elasticsearch with every node running on the single
server.
It has elasticsearch master instance, which balances the load between two
elasticsearch data instances (6 shards and 1 replica per index).
Sometimes I see the following error in the log:

[2013-08-25 12:30:29,549][WARN ][cluster.action.shard ] [es_master]
received shard failed for [logstash-2013.08.25][4],
node[x2u8Oq70T4iELaMs_9FFUA], [R], s[STARTED], reason [Failed to perform
[index] on replica, message
[RemoteTransportException[[es_node2][inet[/192.168.2.221:9301]][index/replica]];
nested: EsRejectedExecutionException[rejected execution of
[org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler]];
]]

It is always the same instance (es_node2) and failed index on replica.
RemoteTransportException seems very strange to me, since es_node2 is on the
same machine with the master and es_node1. And I have never seen such error
for es_node1 (settings for es_node1 and es_node2 are the same). I'm
indexing about 3,5K events per second, so I have socket buffers memory
increased.

Please explain me what does "EsRejectedExecutionException[rejected
execution of
[org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler]]"
mean?

Any help would be highly appreciated.

Thanks!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Hi Gennady,

Did you get anywhere with this issue? I have the same issue.

I have tried setting threadpool.index.queue_size=-1 on ES 0.20.x as
described on
http://jontai.me/blog/2013/06/esrejectedexecutionexception-rejected-execution-of-messagechannelhandler-requesthandler/
but no luck so far.

Regards,
Renzo

Op zondag 25 augustus 2013 10:52:20 UTC+2 schreef Gennady Alexandrov:

Hello!

I have develop setup of elasticsearch with every node running on the
single server.
It has elasticsearch master instance, which balances the load between two
elasticsearch data instances (6 shards and 1 replica per index).
Sometimes I see the following error in the log:

[2013-08-25 12:30:29,549][WARN ][cluster.action.shard ] [es_master]
received shard failed for [logstash-2013.08.25][4],
node[x2u8Oq70T4iELaMs_9FFUA], [R], s[STARTED], reason [Failed to perform
[index] on replica, message
[RemoteTransportException[[es_node2][inet[/192.168.2.221:9301]][index/replica]];
nested: EsRejectedExecutionException[rejected execution of
[org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler]];
]]

It is always the same instance (es_node2) and failed index on replica.
RemoteTransportException seems very strange to me, since es_node2 is on
the same machine with the master and es_node1. And I have never seen such
error for es_node1 (settings for es_node1 and es_node2 are the same). I'm
indexing about 3,5K events per second, so I have socket buffers memory
increased.

Please explain me what does "EsRejectedExecutionException[rejected
execution of
[org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler]]"
mean?

Any help would be highly appreciated.

Thanks!

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Hi!

Yes, I've managed with this.
Setting threadpool.index.queue_size=-1 wouldn't fix it in 0.20.6. I have to
set large value explicitly. In my case threadpool.index.queue_size: 150000
is fine.

2013/9/17 R. Toma renzo.toma@gmail.com

Hi Gennady,

Did you get anywhere with this issue? I have the same issue.

I have tried setting threadpool.index.queue_size=-1 on ES 0.20.x as
described on
http://jontai.me/blog/2013/06/esrejectedexecutionexception-rejected-execution-of-messagechannelhandler-requesthandler/but no luck so far.

Regards,
Renzo

Op zondag 25 augustus 2013 10:52:20 UTC+2 schreef Gennady Alexandrov:

Hello!

I have develop setup of elasticsearch with every node running on the
single server.
It has elasticsearch master instance, which balances the load between two
elasticsearch data instances (6 shards and 1 replica per index).
Sometimes I see the following error in the log:

[2013-08-25 12:30:29,549][WARN ][cluster.action.shard ] [es_master]
received shard failed for [logstash-2013.08.25][4],
node[x2u8Oq70T4iELaMs_9FFUA], [R], s[STARTED], reason [Failed to perform
[index] on replica, message [RemoteTransportException[[es_**node2][inet[/
192.168.2.221:**9301]][index/replica]]; nested:
EsRejectedExecutionException[**rejected execution of
[org.elasticsearch.transport.**netty.MessageChannelHandler$**RequestHandler]];
]]

It is always the same instance (es_node2) and failed index on replica.
RemoteTransportException seems very strange to me, since es_node2 is on
the same machine with the master and es_node1. And I have never seen such
error for es_node1 (settings for es_node1 and es_node2 are the same). I'm
indexing about 3,5K events per second, so I have socket buffers memory
increased.

Please explain me what does "EsRejectedExecutionException[**rejected
execution of [org.elasticsearch.transport.*netty.MessageChannelHandler$
*RequestHandler]]" mean?

Any help would be highly appreciated.

Thanks!

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/9lyPyvpeTSY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Best regards,*
*
Gennady Aleksandrov

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

Hi,

Aaaah, that explains it :slight_smile: Will try with large value.

Thank you!

Regards,
Renzo

Op dinsdag 17 september 2013 14:40:05 UTC+2 schreef Gennady Alexandrov:

Hi!

Yes, I've managed with this.
Setting threadpool.index.queue_size=-1 wouldn't fix it in 0.20.6. I have
to set large value explicitly. In my case threadpool.index.queue_size:
150000 is fine.

2013/9/17 R. Toma <renzo...@gmail.com <javascript:>>

Hi Gennady,

Did you get anywhere with this issue? I have the same issue.

I have tried setting threadpool.index.queue_size=-1 on ES 0.20.x as
described on
http://jontai.me/blog/2013/06/esrejectedexecutionexception-rejected-execution-of-messagechannelhandler-requesthandler/but no luck so far.

Regards,
Renzo

Op zondag 25 augustus 2013 10:52:20 UTC+2 schreef Gennady Alexandrov:

Hello!

I have develop setup of elasticsearch with every node running on the
single server.
It has elasticsearch master instance, which balances the load between
two elasticsearch data instances (6 shards and 1 replica per index).
Sometimes I see the following error in the log:

[2013-08-25 12:30:29,549][WARN ][cluster.action.shard ] [es_master]
received shard failed for [logstash-2013.08.25][4],
node[x2u8Oq70T4iELaMs_9FFUA], [R], s[STARTED], reason [Failed to perform
[index] on replica, message [RemoteTransportException[[es_**
node2][inet[/192.168.2.221:**9301]][index/replica]]; nested:
EsRejectedExecutionException[**rejected execution of
[org.elasticsearch.transport.**netty.MessageChannelHandler$**RequestHandler]];
]]

It is always the same instance (es_node2) and failed index on replica.
RemoteTransportException seems very strange to me, since es_node2 is on
the same machine with the master and es_node1. And I have never seen such
error for es_node1 (settings for es_node1 and es_node2 are the same). I'm
indexing about 3,5K events per second, so I have socket buffers memory
increased.

Please explain me what does "EsRejectedExecutionException[**rejected
execution of [org.elasticsearch.transport.**netty.MessageChannelHandler$
**RequestHandler]]" mean?

Any help would be highly appreciated.

Thanks!

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/9lyPyvpeTSY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
Best regards,*
*
Gennady Aleksandrov

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

Good luck!

2013/9/17 R. Toma renzo.toma@gmail.com

Hi,

Aaaah, that explains it :slight_smile: Will try with large value.

Thank you!

Regards,
Renzo

Op dinsdag 17 september 2013 14:40:05 UTC+2 schreef Gennady Alexandrov:

Hi!

Yes, I've managed with this.
Setting threadpool.index.queue_size=-1 wouldn't fix it in 0.20.6. I have
to set large value explicitly. In my case threadpool.index.queue_size:
150000 is fine.

2013/9/17 R. Toma renzo...@gmail.com

Hi Gennady,

Did you get anywhere with this issue? I have the same issue.

I have tried setting threadpool.index.queue_size=-1 on ES 0.20.x as
described on http://jontai.me/blog/2013/06/**
esrejectedexecutionexception-rejected-execution-of-
messagechannelhandler-**requesthandler/http://jontai.me/blog/2013/06/esrejectedexecutionexception-rejected-execution-of-messagechannelhandler-requesthandler/but no luck so far.

Regards,
Renzo

Op zondag 25 augustus 2013 10:52:20 UTC+2 schreef Gennady Alexandrov:

Hello!

I have develop setup of elasticsearch with every node running on the
single server.
It has elasticsearch master instance, which balances the load between
two elasticsearch data instances (6 shards and 1 replica per index).
Sometimes I see the following error in the log:

[2013-08-25 12:30:29,549][WARN ][cluster.action.shard ] [es_master]
received shard failed for [logstash-2013.08.25][4],
node[x2u8Oq70T4iELaMs_9FFUA], [R], s[STARTED], reason [Failed to perform
[index] on replica, message [RemoteTransportException[[es_****
node2][inet[/192.168.2.221:9301]][index/replica]]; nested:
EsRejectedExecutionException[rejected execution of
[org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler]];
]]

It is always the same instance (es_node2) and failed index on replica.
RemoteTransportException seems very strange to me, since es_node2 is on
the same machine with the master and es_node1. And I have never seen such
error for es_node1 (settings for es_node1 and es_node2 are the same). I'm
indexing about 3,5K events per second, so I have socket buffers memory
increased.

Please explain me what does "EsRejectedExecutionException[****rejected
execution of [org.elasticsearch.transport.n
etty.MessageChannelHandler$RequestHandler]]" mean?

Any help would be highly appreciated.

Thanks!

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/**
topic/elasticsearch/**9lyPyvpeTSY/unsubscribehttps://groups.google.com/d/topic/elasticsearch/9lyPyvpeTSY/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearc...@**googlegroups.com.
For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
Best regards,*
*
Gennady Aleksandrov

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/9lyPyvpeTSY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
Best regards,*
*
Gennady Aleksandrov

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.