# Request rejected when multiple search request **URL:** \ **Category:** Elasticsearch\ **Created:** [August 30, 2019, 8:53am UTC](https://discuss.elastic.co/t/request-rejected-when-multiple-search-request/197476 "2019-08-30T08:53:18Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![bobin](https://avatars.discourse-cdn.com/v4/letter/b/8e7dd6/32.png) [@bobin](https://discuss.elastic.co/u/bobin)\ **Post date:** [August 30, 2019, 8:53am UTC](https://discuss.elastic.co/t/request-rejected-when-multiple-search-request/197476/1 "2019-08-30T08:53:18Z")
Hi Guys, I am trying to call multiple(300) elasticsearch search request Async and I got the following error. It seems like some of my requests being rejected. Anyone experience this before. I am still new to Elasticsearch, please correct me if any wrong. Thanks > 2019-08-30T16:21:32,588][DEBUG][o.e.a.s.TransportSearchAction] [Windows-PC] [data][4], node[tYdfN-D-REye4n7029yCQw], [P], s[STARTED], a[id=z4waoy-VRwuXqO3x6XOypQ]: Failed to execute [SearchRequest{searchType=QUERY\_THEN\_FETCH, indices=[log], indicesOptions=IndicesOptions[ignore\_unavailable=false, allow\_no\_indices=true, expand\_wildcards\_open=true, expand\_wildcards\_closed=false, allow\_aliases\_to\_multiple\_indices=true, forbid\_closed\_indices=true, ignore\_aliases=false, ignore\_throttled=true], types=, routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=128, allowPartialSearchResults=true, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=true, source={"query":{"query\_string":{"query":"data.id:3","fields":,"type":"best\_fields","default\_operator":"or","max\_determinized\_states":10000,"enable\_position\_increments":true,"fuzziness":"AUTO","fuzzy\_prefix\_length":0,"fuzzy\_max\_expansions":50,"phrase\_slop":0,"analyze\_wildcard":false,"escape":false,"auto\_generate\_synonyms\_phrase\_query":true,"fuzzy\_transpositions":true,"boost":1.0}}}}] lastShard [true] > org.elasticsearch.transport.RemoteTransportException: [Windows-PC][127.0.0.1:9300][indices:data/read/search[phase/query]] > Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of org.elasticsearch.common.util.concurrent.TimedRunnable@7a92c500 on QueueResizingEsThreadPoolExecutor[name = WIN10-PC/search, queue capacity = 1000, min queue capacity = 1000, max queue capacity = 2000, frame size = 2000, targeted response rate = 1s, task execution EWMA = 41.4ms, adjustment amount = 50, org.elasticsearch.common.util.concurrent.QueueResizingEsThreadPoolExecutor@538a0253[Running, pool size = 4, active threads = 4, queued tasks = 1000, completed tasks = 5766]] > at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:48) ~[elasticsearch-7.1.1.jar:7.1.1] > at java.util.concurrent.ThreadPoolExecutor.reject(Unknown Source) ~[?:?] > at java.util.concurrent.ThreadPoolExecutor.execute(Unknown Source) ~[?:?] > at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.execute(EsThreadPoolExecutor.java:98) [elasticsearch-7.1.1.jar:7.1.1] > at org.elasticsearch.search.SearchService.lambda$rewriteShardRequest$4(SearchService.java:1061) [elasticsearch-7.1.1.jar:7.1.1] > at org.elasticsearch.index.shard.IndexShard.awaitShardSearchActive(IndexShard.java:3024) [elasticsearch-7.1.1.jar:7.1.1] > at org.elasticsearch.search.SearchService.lambda$rewriteShardRequest$5(SearchService.java:1060) [elasticsearch-7.1.1.jar:7.1.1] > at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) [elasticsearch-7.1.1.jar:7.1.1] > at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:114) [elasticsearch-7.1.1.jar:7.1.1] > at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:87) [elasticsearch-7.1.1.jar:7.1.1] > at org.elasticsearch.search.SearchService.rewriteShardRequest(SearchService.java:1076) [elasticsearch-7.1.1.jar:7.1.1] > at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:354) [elasticsearch-7.1.1.jar:7.1.1] > at org.elasticsearch.action.search.SearchTransportService.lambda$registerRequestHandler$7(SearchTransportService.java:352) [elasticsearch-7.1.1.jar:7.1.1] > at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [elasticsearch-7.1.1.jar:7.1.1] > at org.elasticsearch.transport.TransportService.sendLocalRequest(TransportService.java:687) [elasticsearch-7.1.1.jar:7.1.1] > at org.elasticsearch.transport.TransportService.access$000(TransportService.java:73) [elasticsearch-7.1.1.jar:7.1.1] > at org.elasticsearch.transport.TransportService$3.sendRequest(TransportService.java:125) [elasticsearch-7.1.1.jar:7.1.1] > at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:633) [elasticsearch-7.1.1.jar:7.1.1] > at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:543) [elasticsearch-7.1.1.jar:7.1.1] > at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:587) [elasticsearch-7.1.1.jar:7.1.1] > at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:578) [elasticsearch-7.1.1.jar:7.1.1] > at org.elasticsearch.action.search.SearchTransportService.sendExecuteQuery(SearchTransportService.java:137) [elasticsearch-7.1.1.jar:7.1.1] > at org.elasticsearch.action.search.SearchQueryThenFetchAsyncAction.executePhaseOnShard(SearchQueryThenFetchAsyncAction.java:54) [elasticsearch-7.1.1.jar:7.1.1] > at org.elasticsearch.action.search.InitialSearchPhase.lambda$performPhaseOnShard$4(InitialSearchPhase.java:259) [elasticsearch-7.1.1.jar:7.1.1] > at org.elasticsearch.action.search.InitialSearchPhase.performPhaseOnShard(InitialSearchPhase.java:294) [elasticsearch-7.1.1.jar:7.1.1] > at org.elasticsearch.action.search.InitialSearchPhase.run(InitialSearchPhase.java:158) [elasticsearch-7.1.1.jar:7.1.1] > at org.elasticsearch.action.search.AbstractSearchAsyncAction.executePhase(AbstractSearchAsyncAction.java:166) [elasticsearch-7.1.1.jar:7.1.1] > at org.elasticsearch.action.search.AbstractSearchAsyncAction.start(AbstractSearchAsyncAction.java:126) [elasticsearch-7.1.1.jar:7.1.1] > at org.elasticsearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:514) [elasticsearch-7.1.1.jar:7.1.1] > at org.elasticsearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:399) [elasticsearch-7.1.1.jar:7.1.1] > at org.elasticsearch.action.search.TransportSearchAction.lambda$doExecute$3(TransportSearchAction.java:211) [elasticsearch-7.1.1.jar:7.1.1] > at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:61) [elasticsearch-7.1.1.jar:7.1.1] > at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:114) ... ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [August 30, 2019, 6:15pm UTC](https://discuss.elastic.co/t/request-rejected-when-multiple-search-request/197476/2 "2019-08-30T18:15:30Z")
Yes, it looks like you are overwhelming the cluster. As it is not able to keep up the queue fills up, after which requests are rejected. Each query uses at least one slot on the queue per shard it queries so it could be that you simply have too many shards to allow all the 300 concurrent requests to be queued. It seems like you are running a single node, but it is not clear how much resources this have. I would therefore recommend starting with a lower concurrency level and gradually increase until you find the limit of the node or simply increase the size of the cluster. ---
**Author:** ![bobin](https://avatars.discourse-cdn.com/v4/letter/b/8e7dd6/32.png) [@bobin](https://discuss.elastic.co/u/bobin)\ **Post date:** [September 2, 2019, 1:40am UTC](https://discuss.elastic.co/t/request-rejected-when-multiple-search-request/197476/3 "2019-09-02T01:40:40Z")
Hi Christian, thanks for explanation. Yes, I am running this with one single node. By the way, what kind of resources you are referring to ? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 2, 2019, 4:41am UTC](https://discuss.elastic.co/t/request-rejected-when-multiple-search-request/197476/4 "2019-09-02T04:41:10Z")
With resouces I am referring to CPU, RAM and IOPS. ---
**Author:** ![bobin](https://avatars.discourse-cdn.com/v4/letter/b/8e7dd6/32.png) [@bobin](https://discuss.elastic.co/u/bobin)\ **Post date:** [September 2, 2019, 6:32am UTC](https://discuss.elastic.co/t/request-rejected-when-multiple-search-request/197476/5 "2019-09-02T06:32:43Z")
Currently I am running single elastic node in server machine with 2 processors and 8GB RAM. If i run 200 search request concurrently, it's working fine. Should i configure setting for the thread pool? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 2, 2019, 7:03am UTC](https://discuss.elastic.co/t/request-rejected-when-multiple-search-request/197476/6 "2019-09-02T07:03:05Z")
How many indices and shards is each query targeting? ---
**Author:** ![bobin](https://avatars.discourse-cdn.com/v4/letter/b/8e7dd6/32.png) [@bobin](https://discuss.elastic.co/u/bobin)\ **Post date:** [September 2, 2019, 7:20am UTC](https://discuss.elastic.co/t/request-rejected-when-multiple-search-request/197476/7 "2019-09-02T07:20:38Z")
1 indices with 5 shards ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 2, 2019, 7:43am UTC](https://discuss.elastic.co/t/request-rejected-when-multiple-search-request/197476/8 "2019-09-02T07:43:37Z")
How large is it? If less than a few Tens of GB, can you try shrinking it down to a single shard? ---
**Author:** ![bobin](https://avatars.discourse-cdn.com/v4/letter/b/8e7dd6/32.png) [@bobin](https://discuss.elastic.co/u/bobin)\ **Post date:** [September 2, 2019, 7:45am UTC](https://discuss.elastic.co/t/request-rejected-when-multiple-search-request/197476/9 "2019-09-02T07:45:33Z")
My index is around 25 GB. Is it needed? Should i configure the threadpool setting in elasticsearch.yml? > thread\_pool.search.queue\_size: > thread\_pool.search.size: > thread\_pool.search.min\_queue\_size: > thread\_pool.search.max\_queue\_size: > thread\_pool.search.auto\_queue\_frame\_size: > thread\_pool.search.target\_response\_time: ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [September 30, 2019, 7:45am UTC](https://discuss.elastic.co/t/request-rejected-when-multiple-search-request/197476/10 "2019-09-30T07:45:41Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.