Elasticsearch problem, search thread pool rejected

Elastic Stack Elasticsearch
1

Hello everyone, I have 1 node of elasticsearch 6.0 that it has about 100 indices. When I query Visual builder dashboard, sometime I got "The request for this panel failed. The aggregations key is missing from the response" error messages follow ..

and I check elasticsearch.log and _cat/thread_pool, I found ..

_> org.elasticsearch.transport.RemoteTransportException: [JjBTUs][10.209.60.191:9300][indices:data/read/search[phase/query]]
> Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution of org.elasticsearch.common.util.concurrent.TimedRunnable@3a03a1ea on QueueResizingEsThreadPoolExecutor[search, queue capacity = 1000, min queue capacity = 1000, max queue capacity = 4000, frame size = 2000, targeted response rate = 1s, task execution EWMA = 807.1micros, adjustment amount = 50, QueueResizingEsThreadPoolExecutor[search, queue capacity = 1000, org.elasticsearch.common.util.concurrent.QueueResizingEsThreadPoolExecutor@4ba69385[Running, pool size = 13, active threads = 13, queued tasks = 1019, completed tasks = 383267]]]
> at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:48) ~[elasticsearch-6.0.0.jar:6.0.0]
> at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) ~[?:1.8.0_151]
> at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) ~[?:1.8.0_151]
> at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.doExecute(EsThreadPoolExecutor.java:94) ~[elasticsearch-6.0.0.jar:6.0.0]
> at org.elasticsearch.common.util.concurrent.QueueResizingEsThreadPoolExecutor.doExecute(QueueResizingEsThreadPoolExecutor.java:95) ~[elasticsearch-6.0.0.jar:6.0.0]
> at org.elasticsearch.common.util.concurrent.EsThreadPoolExecutor.execute(EsThreadPoolExecutor.java:89) ~[elasticsearch-6.0.0.jar:6.0.0]
> at org.elasticsearch.search.SearchService.lambda$rewriteShardRequest$0(SearchService.java:956) ~[elasticsearch-6.0.0.jar:6.0.0]
> at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:59) ~[elasticsearch-6.0.0.jar:6.0.0]
> at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:113) ~[elasticsearch-6.0.0.jar:6.0.0]
> at org.elasticsearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:86) ~[elasticsearch-6.0.0.jar:6.0.0]
> at org.elasticsearch.search.SearchService.rewriteShardRequest(SearchService.java:954) ~[elasticsearch-6.0.0.jar:6.0.0]
> at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:284) ~[elasticsearch-6.0.0.jar:6.0.0]
> at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:356) ~[elasticsearch-6.0.0.jar:6.0.0]
> at org.elasticsearch.action.search.SearchTransportService$6.messageReceived(SearchTransportService.java:353) ~[elasticsearch-6.0.0.jar:6.0.0]
> at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:66) ~[elasticsearch-6.0.0.jar:6.0.0]
> at org.elasticsearch.transport.TransportService.sendLocalRequest(TransportService.java:653) ~[elasticsearch-6.0.0.jar:6.0.0]
> at org.elasticsearch.transport.TransportService.access$000(TransportService.java:74) ~[elasticsearch-6.0.0.jar:6.0.0]
> at org.elasticsearch.transport.TransportService$3.sendRequest(TransportService.java:137) ~[elasticsearch-6.0.0.jar:6.0.0]
> at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:601) ~[elasticsearch-6.0.0.jar:6.0.0]
> at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:521) ~[elasticsearch-6.0.0.jar:6.0.0]
> at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:561) ~[elasticsearch-6.0.0.jar:6.0.0]
> at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:552) ~[elasticsearch-6.0.0.jar:6.0.0]
> at org.elasticsearch.action.search.SearchTransportService.sendExecuteQuery(SearchTransportService.java:146) ~[elasticsearch-6.0.0.jar:6.0.0]
> at org.elasticsearch.action.search.SearchQueryThenFetchAsyncAction.executePhaseOnShard(SearchQueryThenFetchAsyncAction.java:52) ~[elasticsearch-6.0.0.jar:6.0.0]
> at org.elasticsearch.action.search.InitialSearchPhase.performPhaseOnShard(InitialSearchPhase.java:208) ~[elasticsearch-6.0.0.jar:6.0.0]
> at org.elasticsearch.action.search.InitialSearchPhase.run(Initi
> :alSearchPhase.java:153) ~[elasticsearch-6.0.0.jar:6.0.0]

Screenshot_1

How can I tune elasticsearch performance for solve this issue ?

  • I already set ..
    thread_pool.search.queue_size: 4000
    thread_pool.search.max_queue_size: 4000
    but it's not work.

Thank you very much :smile:
BR,
CS.

2

Hi @chayapon1219,

Usually, when it happens to the search queue to fill up it means elasticsearch is not having the needed throughput to attend all your requests. Increasing the search queue only makes the problem happens a little later. It's a producer/consumer problem.

To solve this you have some alternatives:

  • Add more resources to your cluster (upgrade hardware, or add more nodes)
  • Optimize your queries
  • Perform fewer requests
  • etc

Hope it helps.

Cheers,
LG

2 Likes
3

How many shards do you have in the cluster? What is the average shard size? How many of these are you typically searching?

1 Like
4

I have about 550 shards with 100+ indices.
average shard size = 8Gb.
but I query these data every 30s (Refresh on dashboard).

Last day,I set ' thread_pool.search.min_queue_size: 2000 ' on elasticsearch.yml and it's work for me. Rejected thread pools are decrease.

Thank you very much for your suggestion.

1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.