# Client side, EsRejectedExecutionException **URL:** \ **Category:** Elasticsearch\ **Created:** [July 23, 2013, 12:07am UTC](https://discuss.elastic.co/t/client-side-esrejectedexecutionexception/12888 "2013-07-23T00:07:57Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Andrew\_Gaydenko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrew_gaydenko/32/914_2.png) [@Andrew\_Gaydenko](https://discuss.elastic.co/u/Andrew_Gaydenko)\ **Post date:** [July 23, 2013, 12:07am UTC](https://discuss.elastic.co/t/client-side-esrejectedexecutionexception/12888/1 "2013-07-23T00:07:57Z")
I have got the exception shown below on ES _client_ side (concurrent environment). Where to dig in? In another group thread I was assured the client itself is thread safe (stateless).So, I guess, I must tune some way execution context. I have tried fixed thread pool. Which another Executor can eliminate the issue? 0.90.2 is in use. Thanks! Exception in thread "main" org.elasticsearch.action.search. SearchPhaseExecutionException: Failed to execute phase [query\_fetch], total failure; shardFailures {RemoteTransportException[[jazz-node][inet[/192.168. 1.101:9301]][search/phase/query+fetch]]; nested: EsRejectedExecutionException[rejected execution of [org.elasticsearch. transport.netty.MessageChannelHandler$RequestHandler]]; } at org.elasticsearch.action.search.type. TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult( TransportSearchTypeAction.java:261) at org.elasticsearch.action.search.type. TransportSearchTypeAction$BaseAsyncAction$3.onFailure( TransportSearchTypeAction.java:214) at org.elasticsearch.search.action.SearchServiceTransportAction$5. handleException(SearchServiceTransportAction.java:263) at org.elasticsearch.transport.netty.MessageChannelHandler. handleException(MessageChannelHandler.java:179) at org.elasticsearch.transport.netty.MessageChannelHandler. handlerResponseError(MessageChannelHandler.java:170) at org.elasticsearch.transport.netty.MessageChannelHandler. messageReceived(MessageChannelHandler.java:122) at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler. handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline. sendUpstream(DefaultChannelPipeline.java:564) at org.elasticsearch.common.netty.channel. DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream( DefaultChannelPipeline.java:791) at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived( Channels.java:296) at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder. unfoldAndFireMessageReceived(FrameDecoder.java:462) at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder. callDecode(FrameDecoder.java:443) at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder. messageReceived(FrameDecoder.java:303) at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler. handleUpstream(SimpleChannelUpstreamHandler.java:70) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline. sendUpstream(DefaultChannelPipeline.java:564) at org.elasticsearch.common.netty.channel.DefaultChannelPipeline. sendUpstream(DefaultChannelPipeline.java:559) at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived( Channels.java:268) at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived( Channels.java:255) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read( NioWorker.java:88) at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker. process(AbstractNioWorker.java:109) at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector .run(AbstractNioSelector.java:312) at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker. run(AbstractNioWorker.java:90) at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run( NioWorker.java:178) at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run( ThreadRenamingRunnable.java:108) at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1. run(DeadLockProofWorker.java:42) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor. java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor .java:615) at java.lang.Thread.run(Thread.java:724) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [July 23, 2013, 12:19am UTC](https://discuss.elastic.co/t/client-side-esrejectedexecutionexception/12888/2 "2013-07-23T00:19:55Z")
Here is a recent blog post that I hope helps: [http://jontai.me/blog/2013/06/esrejectedexecutionexception-rejected-execution-of-messagechannelhandler-requesthandler/](http://jontai.me/blog/2013/06/esrejectedexecutionexception-rejected-execution-of-messagechannelhandler-requesthandler/) -- Ivan On Mon, Jul 22, 2013 at 5:07 PM, Andrew Gaydenko [andrew.gaydenko@gmail.com](mailto:andrew.gaydenko@gmail.com)wrote: > I have got the exception shown below on ES _client_ side (concurrent > environment). Where to dig in? In another group thread I was assured the > client itself is thread safe (stateless).So, I guess, I must tune some way > execution context. I have tried fixed thread pool. Which another Executor > can eliminate the issue? 0.90.2 is in use. Thanks! > > Exception in thread "main" org.elasticsearch.action.search. > SearchPhaseExecutionException: Failed to execute phase [query\_fetch],total failure > ; shardFailures {RemoteTransportException[[jazz-node][inet[/192.168.1.101: > 9301]][search/phase/query+fetch]]; nested: EsRejectedExecutionException[rejected > execution of [org.elasticsearch.transport.netty. > MessageChannelHandler$RequestHandler]]; } > at org.elasticsearch.action.search.type. > TransportSearchTypeAction$BaseAsyncAction.onFirstPhaseResult( > TransportSearchTypeAction.java:261) > at org.elasticsearch.action.search.type. > TransportSearchTypeAction$BaseAsyncAction$3.onFailure( > TransportSearchTypeAction.java:214) > at org.elasticsearch.search.action.SearchServiceTransportAction$5. > handleException(SearchServiceTransportAction.java:263) > at org.elasticsearch.transport.netty.MessageChannelHandler. > handleException(MessageChannelHandler.java:179) > at org.elasticsearch.transport.netty.MessageChannelHandler. > handlerResponseError(MessageChannelHandler.java:170) > at org.elasticsearch.transport.netty.MessageChannelHandler. > messageReceived(MessageChannelHandler.java:122) > at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler > .handleUpstream(SimpleChannelUpstreamHandler.java:70) > at org.elasticsearch.common.netty.channel.DefaultChannelPipeline. > sendUpstream(DefaultChannelPipeline.java:564) > at org.elasticsearch.common.netty.channel. > DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream( > DefaultChannelPipeline.java:791) > at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived > (Channels.java:296) > at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder. > unfoldAndFireMessageReceived(FrameDecoder.java:462) > at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder. > callDecode(FrameDecoder.java:443) > at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder. > messageReceived(FrameDecoder.java:303) > at org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler > .handleUpstream(SimpleChannelUpstreamHandler.java:70) > at org.elasticsearch.common.netty.channel.DefaultChannelPipeline. > sendUpstream(DefaultChannelPipeline.java:564) > at org.elasticsearch.common.netty.channel.DefaultChannelPipeline. > sendUpstream(DefaultChannelPipeline.java:559) > at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived > (Channels.java:268) > at org.elasticsearch.common.netty.channel.Channels.fireMessageReceived > (Channels.java:255) > at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read( > NioWorker.java:88) > at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker > .process(AbstractNioWorker.java:109) > at org.elasticsearch.common.netty.channel.socket.nio. > AbstractNioSelector.run(AbstractNioSelector.java:312) > at org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker > .run(AbstractNioWorker.java:90) > at org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run( > NioWorker.java:178) > at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run( > ThreadRenamingRunnable.java:108) > at org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1. > run(DeadLockProofWorker.java:42) > at java.util.concurrent.ThreadPoolExecutor.runWorker( > ThreadPoolExecutor.java:1145) > at java.util.concurrent.ThreadPoolExecutor$Worker.run( > ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:724) > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Andrew\_Gaydenko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrew_gaydenko/32/914_2.png) [@Andrew\_Gaydenko](https://discuss.elastic.co/u/Andrew_Gaydenko)\ **Post date:** [July 23, 2013, 12:27am UTC](https://discuss.elastic.co/t/client-side-esrejectedexecutionexception/12888/3 "2013-07-23T00:27:50Z")
On Tuesday, July 23, 2013 4:19:55 AM UTC+4, Ivan Brusic wrote: > Here is a recent blog post that I hope helps: > > [http://jontai.me/blog/2013/06/esrejectedexecutionexception-rejected-execution-of-messagechannelhandler-requesthandler/](http://jontai.me/blog/2013/06/esrejectedexecutionexception-rejected-execution-of-messagechannelhandler-requesthandler/) Thanks. Yes, I have found that blog post, but it is related to server size. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Andrew\_Gaydenko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrew_gaydenko/32/914_2.png) [@Andrew\_Gaydenko](https://discuss.elastic.co/u/Andrew_Gaydenko)\ **Post date:** [July 23, 2013, 12:50am UTC](https://discuss.elastic.co/t/client-side-esrejectedexecutionexception/12888/4 "2013-07-23T00:50:31Z")
On Tuesday, July 23, 2013 4:19:55 AM UTC+4, Ivan Brusic wrote: > Here is a recent blog post that I hope helps: > > [http://jontai.me/blog/2013/06/esrejectedexecutionexception-rejected-execution-of-messagechannelhandler-requesthandler/](http://jontai.me/blog/2013/06/esrejectedexecutionexception-rejected-execution-of-messagechannelhandler-requesthandler/) > > -- > Ivan OK, I have tried that suggestion on client size this way (it is Scala code, but I'm sure it is readable): val settings = ImmutableSettings.builder.put("threadpool.search.queue\_size", -1).build val nativeClient = nodeBuilder.client(true).settings(settings).clusterName( clusterName).node.client Nothing changed unfortunately... Highly concurrent indexing works perfectly in the same environment. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![vinh](https://avatars.discourse-cdn.com/v4/letter/v/34f0e0/32.png) [@vinh](https://discuss.elastic.co/u/vinh)\ **Post date:** [July 23, 2013, 1:39am UTC](https://discuss.elastic.co/t/client-side-esrejectedexecutionexception/12888/5 "2013-07-23T01:39:59Z")
This is a server side setting. So it needs to be configured in elasticsearch.yml. -Vinh On Jul 22, 2013, at 5:50 PM, Andrew Gaydenko [andrew.gaydenko@gmail.com](mailto:andrew.gaydenko@gmail.com) wrote: > On Tuesday, July 23, 2013 4:19:55 AM UTC+4, Ivan Brusic wrote: > Here is a recent blog post that I hope helps: > > [http://jontai.me/blog/2013/06/esrejectedexecutionexception-rejected-execution-of-messagechannelhandler-requesthandler/](http://jontai.me/blog/2013/06/esrejectedexecutionexception-rejected-execution-of-messagechannelhandler-requesthandler/) > > -- > Ivan > > OK, I have tried that suggestion on client size this way (it is Scala code, but I'm sure it is readable): > > val settings = ImmutableSettings.builder.put("threadpool.search.queue\_size", -1).build > val nativeClient = nodeBuilder.client(true).settings(settings).clusterName(clusterName).node.client > > Nothing changed unfortunately... > Highly concurrent indexing works perfectly in the same environment. > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Andrew\_Gaydenko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrew_gaydenko/32/914_2.png) [@Andrew\_Gaydenko](https://discuss.elastic.co/u/Andrew_Gaydenko)\ **Post date:** [July 23, 2013, 7:11am UTC](https://discuss.elastic.co/t/client-side-esrejectedexecutionexception/12888/6 "2013-07-23T07:11:10Z")
On Tuesday, July 23, 2013 5:39:59 AM UTC+4, vinh wrote: > This is a server side setting. So it needs to be configured in > elasticsearch.yml. > -Vinh I have set queue size to -1 on server side also without any behavior change. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [July 23, 2013, 4:23pm UTC](https://discuss.elastic.co/t/client-side-esrejectedexecutionexception/12888/7 "2013-07-23T16:23:23Z")
Is there a reason why you are creating your own client instead of using the TransportClient? The defaults have changed in 0.90 and so far I have not had a need to change them. Is the problem consistent? What happens if you change the reject\_policy to caller? -- Ivan On Mon, Jul 22, 2013 at 6:39 PM, vinh [vinh@loggly.com](mailto:vinh@loggly.com) wrote: > This is a server side setting. So it needs to be configured in > elasticsearch.yml. > -Vinh > > On Jul 22, 2013, at 5:50 PM, Andrew Gaydenko [andrew.gaydenko@gmail.com](mailto:andrew.gaydenko@gmail.com) > wrote: > > On Tuesday, July 23, 2013 4:19:55 AM UTC+4, Ivan Brusic wrote: > > > Here is a recent blog post that I hope helps: > > > > [http://jontai.me/blog/2013/06/\*\*esrejectedexecutionexception-](http://jontai.me/blog/2013/06/**esrejectedexecutionexception-)\*\* > > rejected-execution-of-\*\*messagechannelhandler-\*\*requesthandler/[http://jontai.me/blog/2013/06/esrejectedexecutionexception-rejected-execution-of-messagechannelhandler-requesthandler/](http://jontai.me/blog/2013/06/esrejectedexecutionexception-rejected-execution-of-messagechannelhandler-requesthandler/) > > > > -- > > Ivan > > OK, I have tried that suggestion on client size this way (it is Scala > code, but I'm sure it is readable): > > val settings = ImmutableSettings.builder.put( > "threadpool.search.queue\_size", -1).build > val nativeClient = nodeBuilder.client(true).settings(settings).clusterName > (clusterName).node.client > > Nothing changed unfortunately... > Highly concurrent indexing works perfectly in the same environment. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Andrew\_Gaydenko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrew_gaydenko/32/914_2.png) [@Andrew\_Gaydenko](https://discuss.elastic.co/u/Andrew_Gaydenko)\ **Post date:** [July 23, 2013, 4:55pm UTC](https://discuss.elastic.co/t/client-side-esrejectedexecutionexception/12888/8 "2013-07-23T16:55:27Z")
On Tuesday, July 23, 2013 8:23:23 PM UTC+4, Ivan Brusic wrote: > Is there a reason why you are creating your own client instead of using > the TransportClient? Frankly speaking, I don't know about the difference between the ways a client is created. > Is the problem consistent? Absolutely. > What happens if you change the reject\_policy to caller? I need more requests to get the exception. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:24am UTC](https://discuss.elastic.co/t/client-side-esrejectedexecutionexception/12888/9 "2017-07-06T02:24:58Z")