# Elasticsearch problem, search thread pool rejected **URL:** \ **Category:** Elasticsearch\ **Created:** [January 8, 2018, 4:00am UTC](https://discuss.elastic.co/t/elasticsearch-problem-search-thread-pool-rejected/114446 "2018-01-08T04:00:10Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![chayapon1219](https://avatars.discourse-cdn.com/v4/letter/c/d26b3c/32.png) [@chayapon1219](https://discuss.elastic.co/u/chayapon1219)\ **Post date:** [January 8, 2018, 4:00am UTC](https://discuss.elastic.co/t/elasticsearch-problem-search-thread-pool-rejected/114446/1 "2018-01-08T04:00:11Z")
Hello everyone, I have 1 node of elasticsearch 6.0 that it has about 100 indices. When I query Visual builder dashboard, sometime I got "The request for this panel failed. The aggregations key is missing from the response" error messages follow .. > [@The aggregations key is missing from the response error in Kibana dashboard](https://discuss.elastic.co/t/the-aggregations-key-is-missing-from-the-response-error-in-kibana-dashboard/114198): > > Hello, We have 1 kibana 6.0 and 1 elasticsearch 6.0 node. When we tried to view our dashboard with the Visual Builder visualize, sometime we got "The request for this panel failed. The aggregations key is missing from the response" error messages. Are there any suggestion and recommendation about this case? This is the example picture for the error in our kibana dashboard. BR, CS. **and I check elasticsearch.log and \_cat/thread\_pool, I found ..** \_\> org.elasticsearch.transport.RemoteTransportException: [_JjBTUs][10.209.60.191:9300][indices:data/read/search[phase/query]]_ _\> Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of org.elasticsearch.common.util.concurrent.TimedRunnable@3a03a1ea on QueueResizingEsThreadPoolExecutor[search, queue capacity = 1000, min queue capacity = 1000, max queue capacity = 4000, frame size = 2000, targeted response rate = 1s, task execution EWMA = 807.1micros, adjustment amount = 50, QueueResizingEsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.QueueResizingEsThreadPoolExecutor@4ba69385[Running, pool size = 13, active threads = 13, queued tasks = 1019, completed tasks = 383267]]]_ _\> at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:48) ~[elasticsearch-6.0.0.jar:6.0.0]_ _\> at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0\_151]_ _\> at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) ~[?:1.8.0\_151]_ _\> at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.doExecute(EsThreadPoolExecutor.java:94) ~[elasticsearch-6.0.0.jar:6.0.0]_ _\> at org.elasticsearch.common.util.concurrent.QueueResizingEsThreadPoolExecutor.doExecute(QueueResizingEsThreadPoolExecutor.java:95) ~[elasticsearch-6.0.0.jar:6.0.0]_ _\> at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.execute(EsThreadPoolExecutor.java:89) ~[elasticsearch-6.0.0.jar:6.0.0]_ _\> at org.elasticsearch.search.SearchService.lambda$rewriteShardRequest$0(SearchService.java:956) ~[elasticsearch-6.0.0.jar:6.0.0]_ _\> at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:59) ~[elasticsearch-6.0.0.jar:6.0.0]_ _\> at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:113) ~[elasticsearch-6.0.0.jar:6.0.0]_ _\> at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:86) ~[elasticsearch-6.0.0.jar:6.0.0]_ _\> at org.elasticsearch.search.SearchService.rewriteShardRequest(SearchService.java:954) ~[elasticsearch-6.0.0.jar:6.0.0]_ _\> at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:284) ~[elasticsearch-6.0.0.jar:6.0.0]_ _\> at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:356) ~[elasticsearch-6.0.0.jar:6.0.0]_ _\> at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:353) ~[elasticsearch-6.0.0.jar:6.0.0]_ _\> at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) ~[elasticsearch-6.0.0.jar:6.0.0]_ _\> at org.elasticsearch.transport.TransportService.sendLocalRequest(TransportService.java:653) ~[elasticsearch-6.0.0.jar:6.0.0]_ _\> at org.elasticsearch.transport.TransportService.access$000(TransportService.java:74) ~[elasticsearch-6.0.0.jar:6.0.0]_ _\> at org.elasticsearch.transport.TransportService$3.sendRequest(TransportService.java:137) ~[elasticsearch-6.0.0.jar:6.0.0]_ _\> at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:601) ~[elasticsearch-6.0.0.jar:6.0.0]_ _\> at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:521) ~[elasticsearch-6.0.0.jar:6.0.0]_ _\> at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:561) ~[elasticsearch-6.0.0.jar:6.0.0]_ _\> at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:552) ~[elasticsearch-6.0.0.jar:6.0.0]_ _\> at org.elasticsearch.action.search.SearchTransportService.sendExecuteQuery(SearchTransportService.java:146) ~[elasticsearch-6.0.0.jar:6.0.0]_ _\> at org.elasticsearch.action.search.SearchQueryThenFetchAsyncAction.executePhaseOnShard(SearchQueryThenFetchAsyncAction.java:52) ~[elasticsearch-6.0.0.jar:6.0.0]_ _\> at org.elasticsearch.action.search.InitialSearchPhase.performPhaseOnShard(InitialSearchPhase.java:208) ~[elasticsearch-6.0.0.jar:6.0.0]_ _\> at org.elasticsearch.action.search.InitialSearchPhase.run(Initi_ _\> :alSearchPhase.java:153) ~[elasticsearch-6.0.0.jar:6.0.0]_ ![Screenshot_1](https://us1.discourse-cdn.com/elastic/original/3X/2/a/2a4f3083d7bc90fb5046627ba1a11936e663b244.jpg) How can I tune elasticsearch performance for solve this issue ? - I already set .. thread\_pool.search.queue\_size: 4000 thread\_pool.search.max\_queue\_size: 4000 but it's not work. Thank you very much 😄 BR, CS. ---
**Author:** ![luiz.santos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/luiz.santos/32/24664_2.png) [@luiz.santos](https://discuss.elastic.co/u/luiz.santos)\ **Post date:** [January 9, 2018, 12:18pm UTC](https://discuss.elastic.co/t/elasticsearch-problem-search-thread-pool-rejected/114446/2 "2018-01-09T12:18:19Z")
Hi @chayapon1219, Usually, when it happens to the search queue to fill up it means elasticsearch is not having the needed throughput to attend all your requests. Increasing the search queue only makes the problem happens a little later. It's a producer/consumer problem. To solve this you have some alternatives: - Add more resources to your cluster (upgrade hardware, or add more nodes) - Optimize your queries - Perform fewer requests - etc Hope it helps. Cheers, LG ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 9, 2018, 12:32pm UTC](https://discuss.elastic.co/t/elasticsearch-problem-search-thread-pool-rejected/114446/3 "2018-01-09T12:32:42Z")
How many shards do you have in the cluster? What is the average shard size? How many of these are you typically searching? ---
**Author:** ![chayapon1219](https://avatars.discourse-cdn.com/v4/letter/c/d26b3c/32.png) [@chayapon1219](https://discuss.elastic.co/u/chayapon1219)\ **Post date:** [January 11, 2018, 4:21am UTC](https://discuss.elastic.co/t/elasticsearch-problem-search-thread-pool-rejected/114446/4 "2018-01-11T04:21:56Z")
I have about 550 shards with 100+ indices. average shard size = 8Gb. but I query these data every 30s (Refresh on dashboard). Last day,I set ' **thread\_pool.search.min\_queue\_size: 2000**' on elasticsearch.yml and it's work for me. Rejected thread pools are decrease. Thank you very much for your suggestion. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 8, 2018, 4:22am UTC](https://discuss.elastic.co/t/elasticsearch-problem-search-thread-pool-rejected/114446/5 "2018-02-08T04:22:03Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.