Es Rejected Exception [es_rejected_execution_exception]

Elastic Stack Elasticsearch
1

I am getting Es rejected Exception.

I am having Elasticsearch setup in a server of 188GB RAM,32 GB is assigned to JVM heap of Elastic search.Remaing is off heap memory.
I am having ~1TB data space.and i am querying on ~200GB data.

My number of queries also not much.
Error:Uncaught Elasticsearch\Common\Exceptions\BadRequest400Exception: {"error":{"root_cause":,"type":"search_phase_execution_exception","reason":"failed to expand hits","phase":"expand","grouped":true,"failed_shards":,"caused_by":{"type":"search_phase_execution_exception","reason":"","phase":"fetch","grouped":true,"failed_shards":,"caused_by":{"type":"es_rejected_execution_exception","reason":"rejected execution of TimedRunnable{original=org.elasticsearch.action.search.FetchSearchPhase$1@69c76223, creationTimeNanos=11618195507137764, startTimeNanos=0, finishTimeNanos=-1, failedOrRejected=false} on TaskExecutionTimeTrackingEsThreadPoolExecutor[name = localhost_es.com/search, queue capacity = 1000, task execution EWMA = 79.9ms, total task execution time = 38.1d, org.elasticsearch.common.util.concurrent.TaskExecutionTimeTrackingEsThreadPoolExecutor@52e34db[Running, pool size = 61, active threads = 61, queued tasks = 1000, completed tasks = 762999813]]"}}},"status":429}

How to solve this..

2

Welcome to the forum.

Is likely your immediate problem. I guess you have no clear idea why there are so many tasks queued? FYI 1000 is the default limit for the thread pool. Is this issue new, and if so has anything changed recently? That default limit was the same last week!

You could have a look at the /_tasks and _nodes/stats/thread_pool outputs. And the slow log. You could also increase the limit. But might be better to understand your use case and query pattern a little more.

3

Which version of Elasticsearch are you using?

What type of storage are you using? Local SSDs?

How much data does the node hold in total? How many indices and shards is that spread across?

How many indices and shards are your queries targeting?

How many concurrent queries are you sending? What type of queries are these?