EsRejectedExecutionException

Elastic Stack Elasticsearch
1

Hi All,

In my cluster, I'm having around 500 indices. When i'm trying to start the
elasticsearch instance, its showing the following exception.
Why its happening, what should be done to resolve it?

Thanks,
Anand

[2014-07-31 11:50:01,551][DEBUG][action.search.type ] [ESCS_NODE]
[components][3], node[VQMkI5CBRB2hDEnZ5yVLcw], [P], s[STARTED]: Failed to
execute [org.elasticsearch.action.search.SearchRequest@c34bbb8] lastShard
[true]
org.elasticsearch.common.util.concurrent.EsRejectedExecutionException:
rejected execution (queue capacity 1000) on
org.elasticsearch.search.action.SearchServiceTransportAction$23@60cfc409
at
org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:62)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
at
org.elasticsearch.search.action.SearchServiceTransportAction.execute(SearchServiceTransportAction.java:509)
at
org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:203)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:171)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.start(TransportSearchTypeAction.java:153)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction.doExecute(TransportSearchQueryThenFetchAction.java:59)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction.doExecute(TransportSearchQueryThenFetchAction.java:49)
at
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:63)
at
org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:101)
at
org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:43)
at
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:63)
at org.elasticsearch.client.node.NodeClient.execute(NodeClient.java:92)
at
org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:212)
at
org.elasticsearch.rest.action.search.RestSearchAction.handleRequest(RestSearchAction.java:75)
at
org.elasticsearch.rest.RestController.executeHandler(RestController.java:159)
at
org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:142)
at
org.elasticsearch.http.HttpServer.internalDispatchRequest(HttpServer.java:121)
at
org.elasticsearch.http.HttpServer$Dispatcher.dispatchRequest(HttpServer.java:83)
at
org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(NettyHttpServerTransport.java:294)
at
org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:44)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.elasticsearch.common.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)
at
org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
at
org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6fffef94-9082-42a8-8ed7-cbabd5d5a795%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

The thread pool will reject any search requests when there are 1000 actions
already queued.

Do you have this many search requests at one time? Do you have warmers
and/or percolators running since you mentioned it occurred at startup?

--
Ivan

On Thu, Jul 31, 2014 at 12:36 AM, Anand kumar anandv1000@gmail.com wrote:

Hi All,

In my cluster, I'm having around 500 indices. When i'm trying to start the
elasticsearch instance, its showing the following exception.
Why its happening, what should be done to resolve it?

Thanks,
Anand

[2014-07-31 11:50:01,551][DEBUG][action.search.type ] [ESCS_NODE]
[components][3], node[VQMkI5CBRB2hDEnZ5yVLcw], [P], s[STARTED]: Failed to
execute [org.elasticsearch.action.search.SearchRequest@c34bbb8] lastShard
[true]
org.elasticsearch.common.util.concurrent.EsRejectedExecutionException:
rejected execution (queue capacity 1000) on
org.elasticsearch.search.action.SearchServiceTransportAction$23@60cfc409
at
org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:62)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
at
org.elasticsearch.search.action.SearchServiceTransportAction.execute(SearchServiceTransportAction.java:509)
at
org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:203)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:171)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.start(TransportSearchTypeAction.java:153)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction.doExecute(TransportSearchQueryThenFetchAction.java:59)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction.doExecute(TransportSearchQueryThenFetchAction.java:49)
at
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:63)
at
org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:101)
at
org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:43)
at
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:63)
at org.elasticsearch.client.node.NodeClient.execute(NodeClient.java:92)
at
org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:212)
at
org.elasticsearch.rest.action.search.RestSearchAction.handleRequest(RestSearchAction.java:75)
at
org.elasticsearch.rest.RestController.executeHandler(RestController.java:159)
at
org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:142)
at
org.elasticsearch.http.HttpServer.internalDispatchRequest(HttpServer.java:121)
at
org.elasticsearch.http.HttpServer$Dispatcher.dispatchRequest(HttpServer.java:83)
at
org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(NettyHttpServerTransport.java:294)
at
org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:44)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.elasticsearch.common.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)
at
org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
at
org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/6fffef94-9082-42a8-8ed7-cbabd5d5a795%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/6fffef94-9082-42a8-8ed7-cbabd5d5a795%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQA8NOHOwDS3ffM-kF40JBU%2Bxi%3D364JAyJ3M%2BN%2B1qRy7zg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

Thank you Ivan.

If so i'm having 500 indices, each splitted into 5 shards. And while
starting the instance, it'll update each shards. In that case, will it
create 500*5 requests?

Also i don't have any warmer or percolator. As percolator are fixed on
processor size, is there any chance of automatic updation on these?

Thanks again for reporting this.

On Thursday, 31 July 2014 22:41:19 UTC+5:30, Ivan Brusic wrote:

The thread pool will reject any search requests when there are 1000
actions already queued.

Elasticsearch Platform — Find real-time answers at scale | Elastic

Do you have this many search requests at one time? Do you have warmers
and/or percolators running since you mentioned it occurred at startup?

--
Ivan

On Thu, Jul 31, 2014 at 12:36 AM, Anand kumar <anand...@gmail.com
<javascript:>> wrote:

Hi All,

In my cluster, I'm having around 500 indices. When i'm trying to start
the elasticsearch instance, its showing the following exception.
Why its happening, what should be done to resolve it?

Thanks,
Anand

[2014-07-31 11:50:01,551][DEBUG][action.search.type ] [ESCS_NODE]
[components][3], node[VQMkI5CBRB2hDEnZ5yVLcw], [P], s[STARTED]: Failed to
execute [org.elasticsearch.action.search.SearchRequest@c34bbb8] lastShard
[true]
org.elasticsearch.common.util.concurrent.EsRejectedExecutionException:
rejected execution (queue capacity 1000) on
org.elasticsearch.search.action.SearchServiceTransportAction$23@60cfc409
at
org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:62)
at
java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
at
java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
at
org.elasticsearch.search.action.SearchServiceTransportAction.execute(SearchServiceTransportAction.java:509)
at
org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:203)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(TransportSearchQueryThenFetchAction.java:80)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:171)
at
org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.start(TransportSearchTypeAction.java:153)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction.doExecute(TransportSearchQueryThenFetchAction.java:59)
at
org.elasticsearch.action.search.type.TransportSearchQueryThenFetchAction.doExecute(TransportSearchQueryThenFetchAction.java:49)
at
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:63)
at
org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:101)
at
org.elasticsearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:43)
at
org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:63)
at
org.elasticsearch.client.node.NodeClient.execute(NodeClient.java:92)
at
org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:212)
at
org.elasticsearch.rest.action.search.RestSearchAction.handleRequest(RestSearchAction.java:75)
at
org.elasticsearch.rest.RestController.executeHandler(RestController.java:159)
at
org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:142)
at
org.elasticsearch.http.HttpServer.internalDispatchRequest(HttpServer.java:121)
at
org.elasticsearch.http.HttpServer$Dispatcher.dispatchRequest(HttpServer.java:83)
at
org.elasticsearch.http.netty.NettyHttpServerTransport.dispatchRequest(NettyHttpServerTransport.java:294)
at
org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(HttpRequestHandler.java:44)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.elasticsearch.common.netty.handler.codec.http.HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:296)
at
org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.unfoldAndFireMessageReceived(FrameDecoder.java:459)
at
org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
at
org.elasticsearch.common.netty.handler.codec.replay.ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
at
org.elasticsearch.common.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:70)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:791)
at
org.elasticsearch.common.netty.OpenChannelsHandler.handleUpstream(OpenChannelsHandler.java:74)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
at
org.elasticsearch.common.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at
org.elasticsearch.common.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.read(NioWorker.java:88)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.process(AbstractNioWorker.java:108)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSelector.java:318)
at
org.elasticsearch.common.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:89)
at
org.elasticsearch.common.netty.channel.socket.nio.NioWorker.run(NioWorker.java:178)
at
org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
at
org.elasticsearch.common.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/6fffef94-9082-42a8-8ed7-cbabd5d5a795%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/6fffef94-9082-42a8-8ed7-cbabd5d5a795%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/075e105b-9140-4e6a-8518-7e8be2481f13%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

Your stack trace indicates that your search pool was exhausted, not your
admin action thread pool. I don't think that restarting servers will cause
searches to execute, but hopefully someone else would chime in and correct
me. Do you have that many searches going on?

--
Ivan

On Fri, Aug 1, 2014 at 5:45 AM, Anand kumar anandv1000@gmail.com wrote:

Thank you Ivan.

If so i'm having 500 indices, each splitted into 5 shards. And while
starting the instance, it'll update each shards. In that case, will it
create 500*5 requests?

Also i don't have any warmer or percolator. As percolator are fixed on
processor size, is there any chance of automatic updation on these?

Thanks again for reporting this.

On Thursday, 31 July 2014 22:41:19 UTC+5:30, Ivan Brusic wrote:

The thread pool will reject any search requests when there are 1000
actions already queued.

Elasticsearch Platform — Find real-time answers at scale | Elastic
reference/current/modules-threadpool.html

Do you have this many search requests at one time? Do you have warmers
and/or percolators running since you mentioned it occurred at startup?

--
Ivan

On Thu, Jul 31, 2014 at 12:36 AM, Anand kumar anand...@gmail.com wrote:

Hi All,

In my cluster, I'm having around 500 indices. When i'm trying to start
the elasticsearch instance, its showing the following exception.
Why its happening, what should be done to resolve it?

Thanks,
Anand

[2014-07-31 11:50:01,551][DEBUG][action.search.type ] [ESCS_NODE]
[components][3], node[VQMkI5CBRB2hDEnZ5yVLcw], [P], s[STARTED]: Failed to
execute [org.elasticsearch.action.search.SearchRequest@c34bbb8]
lastShard [true]
org.elasticsearch.common.util.concurrent.EsRejectedExecutionException:
rejected execution (queue capacity 1000) on org.elasticsearch.search.
action.SearchServiceTransportAction$23@60cfc409
at org.elasticsearch.common.util.concurrent.EsAbortPolicy.
rejectedExecution(EsAbortPolicy.java:62)
at java.util.concurrent.ThreadPoolExecutor.reject(
ThreadPoolExecutor.java:821)
at java.util.concurrent.ThreadPoolExecutor.execute(
ThreadPoolExecutor.java:1372)
at org.elasticsearch.search.action.SearchServiceTransportAction.
execute(SearchServiceTransportAction.java:509)
at org.elasticsearch.search.action.SearchServiceTransportAction.
sendExecuteQuery(SearchServiceTransportAction.java:203)
at org.elasticsearch.action.search.type.
TransportSearchQueryThenFetchAction$AsyncAction.sendExecuteFirstPhase(
TransportSearchQueryThenFetchAction.java:80)
at org.elasticsearch.action.search.type.TransportSearchTypeAction$
BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:171)
at org.elasticsearch.action.search.type.TransportSearchTypeAction$
BaseAsyncAction.start(TransportSearchTypeAction.java:153)
at org.elasticsearch.action.search.type.
TransportSearchQueryThenFetchAction.doExecute(
TransportSearchQueryThenFetchAction.java:59)
at org.elasticsearch.action.search.type.
TransportSearchQueryThenFetchAction.doExecute(
TransportSearchQueryThenFetchAction.java:49)
at org.elasticsearch.action.support.TransportAction.
execute(TransportAction.java:63)
at org.elasticsearch.action.search.TransportSearchAction.doExecute(
TransportSearchAction.java:101)
at org.elasticsearch.action.search.TransportSearchAction.doExecute(
TransportSearchAction.java:43)
at org.elasticsearch.action.support.TransportAction.
execute(TransportAction.java:63)
at org.elasticsearch.client.node.NodeClient.execute(NodeClient.
java:92)
at org.elasticsearch.client.support.AbstractClient.search(
AbstractClient.java:212)
at org.elasticsearch.rest.action.search.RestSearchAction.
handleRequest(RestSearchAction.java:75)
at org.elasticsearch.rest.RestController.executeHandler(
RestController.java:159)
at org.elasticsearch.rest.RestController.dispatchRequest(
RestController.java:142)
at org.elasticsearch.http.HttpServer.internalDispatchRequest(
HttpServer.java:121)
at org.elasticsearch.http.HttpServer$Dispatcher.
dispatchRequest(HttpServer.java:83)
at org.elasticsearch.http.netty.NettyHttpServerTransport.
dispatchRequest(NettyHttpServerTransport.java:294)
at org.elasticsearch.http.netty.HttpRequestHandler.messageReceived(
HttpRequestHandler.java:44)
at org.elasticsearch.common.netty.channel.
SimpleChannelUpstreamHandler.handleUpstream(
SimpleChannelUpstreamHandler.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.
sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$
DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:
791)
at org.elasticsearch.common.netty.handler.codec.http.
HttpChunkAggregator.messageReceived(HttpChunkAggregator.java:145)
at org.elasticsearch.common.netty.channel.
SimpleChannelUpstreamHandler.handleUpstream(
SimpleChannelUpstreamHandler.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.
sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$
DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:
791)
at org.elasticsearch.common.netty.channel.Channels.
fireMessageReceived(Channels.java:296)
at org.elasticsearch.common.netty.handler.codec.frame.FrameDecoder.
unfoldAndFireMessageReceived(FrameDecoder.java:459)
at org.elasticsearch.common.netty.handler.codec.replay.
ReplayingDecoder.callDecode(ReplayingDecoder.java:536)
at org.elasticsearch.common.netty.handler.codec.replay.
ReplayingDecoder.messageReceived(ReplayingDecoder.java:435)
at org.elasticsearch.common.netty.channel.
SimpleChannelUpstreamHandler.handleUpstream(
SimpleChannelUpstreamHandler.java:70)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.
sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline$
DefaultChannelHandlerContext.sendUpstream(DefaultChannelPipeline.java:
791)
at org.elasticsearch.common.netty.OpenChannelsHandler.
handleUpstream(OpenChannelsHandler.java:74)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.
sendUpstream(DefaultChannelPipeline.java:564)
at org.elasticsearch.common.netty.channel.DefaultChannelPipeline.
sendUpstream(DefaultChannelPipeline.java:559)
at org.elasticsearch.common.netty.channel.Channels.
fireMessageReceived(Channels.java:268)
at org.elasticsearch.common.netty.channel.Channels.
fireMessageReceived(Channels.java:255)
at org.elasticsearch.common.netty.channel.socket.nio.
NioWorker.read(NioWorker.java:88)
at org.elasticsearch.common.netty.channel.socket.nio.
AbstractNioWorker.process(AbstractNioWorker.java:108)
at org.elasticsearch.common.netty.channel.socket.nio.
AbstractNioSelector.run(AbstractNioSelector.java:318)
at org.elasticsearch.common.netty.channel.socket.nio.
AbstractNioWorker.run(AbstractNioWorker.java:89)
at org.elasticsearch.common.netty.channel.socket.nio.
NioWorker.run(NioWorker.java:178)
at org.elasticsearch.common.netty.util.ThreadRenamingRunnable.run(
ThreadRenamingRunnable.java:108)
at org.elasticsearch.common.netty.util.internal.
DeadLockProofWorker$1.run(DeadLockProofWorker.java:42)
at java.util.concurrent.ThreadPoolExecutor.runWorker(
ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(
ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/6fffef94-9082-42a8-8ed7-cbabd5d5a795%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/6fffef94-9082-42a8-8ed7-cbabd5d5a795%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/075e105b-9140-4e6a-8518-7e8be2481f13%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/075e105b-9140-4e6a-8518-7e8be2481f13%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CALY%3DcQA5rUhB7vG%2BT6uOzAHdTodsPK4%3DcYMPg7aTkvTsiWPb6g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.