Client side, EsRejectedExecutionException

Elastic Stack Elasticsearch
1

I have got the exception shown below on ES client side (concurrent
environment). Where to dig in? In another group thread I was assured the
client itself is thread safe (stateless).So, I guess, I must tune some way
execution context. I have tried fixed thread pool. Which another Executor
can eliminate the issue? 0.90.2 is in use. Thanks!

Exception in thread "main" org.elasticsearch.action.search.
SearchPhaseExecutionException: Failed to execute phase [query_fetch], total
failure; shardFailures {RemoteTransportException[[jazz-node][inet[/192.168.
1.101:9301]][search/phase/query+fetch]]; nested:
EsRejectedExecutionException[rejected execution of [org.elasticsearch.
transport.netty.MessageChannelHandler$RequestHandler]]; }
at org.elasticsearch.action.search.type.
TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(
TransportSearchTypeAction.java:261)
at org.elasticsearch.action.search.type.
TransportSearchTypeAction$BaseAsyncAction$3.onFailure(
TransportSearchTypeAction.java:214)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.
handleException(SearchServiceTransportAction.java:263)
at org.elasticsearch.transport.netty.MessageChannelHandler.
handleException(MessageChannelHandler.java:179)
at org.elasticsearch.transport.netty.MessageChannelHandler.
handlerResponseError(MessageChannelHandler.java:170)
at org.elasticsearch.transport.netty.MessageChannelHandler.
messageReceived(MessageChannelHandler.java:122)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.
handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.
sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.
DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(
DefaultChannelPipeline.java:791)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(
Channels.java:296)
at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.
unfoldAndFireMessageReceived(FrameDecoder.java:462)
at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.
callDecode(FrameDecoder.java:443)
at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.
messageReceived(FrameDecoder.java:303)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.
handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.
sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.
sendUpstream(DefaultChannelPipeline.java:559)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(
Channels.java:268)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(
Channels.java:255)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(
NioWorker.java:88)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.
process(AbstractNioWorker.java:109)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector
.run(AbstractNioSelector.java:312)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.
run(AbstractNioWorker.java:90)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(
NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(
ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.
run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:615)
at java.lang.Thread.run(Thread.java:724)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Here is a recent blog post that I hope helps:

http://jontai.me/blog/2013/06/esrejectedexecutionexception-rejected-execution-of-messagechannelhandler-requesthandler/

--
Ivan

On Mon, Jul 22, 2013 at 5:07 PM, Andrew Gaydenko
andrew.gaydenko@gmail.comwrote:

I have got the exception shown below on ES client side (concurrent
environment). Where to dig in? In another group thread I was assured the
client itself is thread safe (stateless).So, I guess, I must tune some way
execution context. I have tried fixed thread pool. Which another Executor
can eliminate the issue? 0.90.2 is in use. Thanks!

Exception in thread "main" org.elasticsearch.action.search.
SearchPhaseExecutionException: Failed to execute phase [query_fetch],total failure
; shardFailures {RemoteTransportException[[jazz-node][inet[/192.168.1.101:
9301]][search/phase/query+fetch]]; nested: EsRejectedExecutionException[rejected
execution of [org.elasticsearch.transport.netty.
MessageChannelHandler$RequestHandler]]; }
at org.elasticsearch.action.search.type.
TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult(
TransportSearchTypeAction.java:261)
at org.elasticsearch.action.search.type.
TransportSearchTypeAction$BaseAsyncAction$3.onFailure(
TransportSearchTypeAction.java:214)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.
handleException(SearchServiceTransportAction.java:263)
at org.elasticsearch.transport.netty.MessageChannelHandler.
handleException(MessageChannelHandler.java:179)
at org.elasticsearch.transport.netty.MessageChannelHandler.
handlerResponseError(MessageChannelHandler.java:170)
at org.elasticsearch.transport.netty.MessageChannelHandler.
messageReceived(MessageChannelHandler.java:122)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler
.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.
sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.
DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(
DefaultChannelPipeline.java:791)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived
(Channels.java:296)
at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.
unfoldAndFireMessageReceived(FrameDecoder.java:462)
at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.
callDecode(FrameDecoder.java:443)
at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.
messageReceived(FrameDecoder.java:303)
at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler
.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.
sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.
sendUpstream(DefaultChannelPipeline.java:559)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived
(Channels.java:268)
at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived
(Channels.java:255)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(
NioWorker.java:88)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker
.process(AbstractNioWorker.java:109)
at org.elasticsearch.common.netty.channel.socket.nio.
AbstractNioSelector.run(AbstractNioSelector.java:312)
at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker
.run(AbstractNioWorker.java:90)
at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(
NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(
ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.
run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

On Tuesday, July 23, 2013 4:19:55 AM UTC+4, Ivan Brusic wrote:

Here is a recent blog post that I hope helps:

http://jontai.me/blog/2013/06/esrejectedexecutionexception-rejected-execution-of-messagechannelhandler-requesthandler/

Thanks. Yes, I have found that blog post, but it is related to server size.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

On Tuesday, July 23, 2013 4:19:55 AM UTC+4, Ivan Brusic wrote:

Here is a recent blog post that I hope helps:

http://jontai.me/blog/2013/06/esrejectedexecutionexception-rejected-execution-of-messagechannelhandler-requesthandler/

--
Ivan

OK, I have tried that suggestion on client size this way (it is Scala code,
but I'm sure it is readable):

val settings = ImmutableSettings.builder.put("threadpool.search.queue_size",
-1).build
val nativeClient = nodeBuilder.client(true).settings(settings).clusterName(
clusterName).node.client

Nothing changed unfortunately...
Highly concurrent indexing works perfectly in the same environment.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

This is a server side setting. So it needs to be configured in elasticsearch.yml.
-Vinh

On Jul 22, 2013, at 5:50 PM, Andrew Gaydenko andrew.gaydenko@gmail.com wrote:

On Tuesday, July 23, 2013 4:19:55 AM UTC+4, Ivan Brusic wrote:
Here is a recent blog post that I hope helps:

http://jontai.me/blog/2013/06/esrejectedexecutionexception-rejected-execution-of-messagechannelhandler-requesthandler/

--
Ivan

OK, I have tried that suggestion on client size this way (it is Scala code, but I'm sure it is readable):

val settings = ImmutableSettings.builder.put("threadpool.search.queue_size", -1).build
val nativeClient = nodeBuilder.client(true).settings(settings).clusterName(clusterName).node.client

Nothing changed unfortunately...
Highly concurrent indexing works perfectly in the same environment.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

6

On Tuesday, July 23, 2013 5:39:59 AM UTC+4, vinh wrote:

This is a server side setting. So it needs to be configured in
elasticsearch.yml.
-Vinh

I have set queue size to -1 on server side also without any behavior change.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

7

Is there a reason why you are creating your own client instead of using the
TransportClient?
The defaults have changed in 0.90 and so far I have not had a need to
change them.

Is the problem consistent? What happens if you change the reject_policy to
caller?

--
Ivan

On Mon, Jul 22, 2013 at 6:39 PM, vinh vinh@loggly.com wrote:

This is a server side setting. So it needs to be configured in
elasticsearch.yml.
-Vinh

On Jul 22, 2013, at 5:50 PM, Andrew Gaydenko andrew.gaydenko@gmail.com
wrote:

On Tuesday, July 23, 2013 4:19:55 AM UTC+4, Ivan Brusic wrote:

Here is a recent blog post that I hope helps:

http://jontai.me/blog/2013/06/**esrejectedexecutionexception-**
rejected-execution-of-**messagechannelhandler-**requesthandler/http://jontai.me/blog/2013/06/esrejectedexecutionexception-rejected-execution-of-messagechannelhandler-requesthandler/

--
Ivan

OK, I have tried that suggestion on client size this way (it is Scala
code, but I'm sure it is readable):

val settings = ImmutableSettings.builder.put(
"threadpool.search.queue_size", -1).build
val nativeClient = nodeBuilder.client(true).settings(settings).clusterName
(clusterName).node.client

Nothing changed unfortunately...
Highly concurrent indexing works perfectly in the same environment.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

8

On Tuesday, July 23, 2013 8:23:23 PM UTC+4, Ivan Brusic wrote:

Is there a reason why you are creating your own client instead of using
the TransportClient?

Frankly speaking, I don't know about the difference between the ways a
client is created.

Is the problem consistent?

Absolutely.

What happens if you change the reject_policy to caller?

I need more requests to get the exception.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.