Rejected execution

[alexus]

Hello World!

I'm using Elasticsearch 6.8.10 and trying to run a query, yet running into following issue:

{
   "shard" : 3,
   "node" : "koEOgkkDBhOVfKHA7QXBjj",
   "reason" : {
      "reason" : "rejected execution of org.elasticsearch.common.util.concurrent.TimedRunnable@26c03d4a on QueueResizingEsThreadPoolExecutor[name = esdX/search, queue capacity = 1000, min queue capacity = 1000, max queue capacity = 1000, frame size = 2000, targeted response rate = 1s, task execution EWMA = 968.1ms, adjustment amount = 50, org.elasticsearch.common.util.concurrent.QueueResizingEsThreadPoolExecutor@70be0765[Running, pool size = 13, active threads = 13, queued tasks = 1000, completed tasks = 616499351]]",
      "type" : "es_rejected_execution_exception"
   },
   "index" : "X-X-X"
}

Please advise.
Thanks in advance!

[warkolm (Mark Walkom)]

Sounds like Elasticsearch is overloaded.
Do you have Monitoring enabled?

[alexus]

• search rate = ~2-3k
• search latency (ms) = ~1ms
• index rate = ~3-4k
• indexing latency (ms) = ~>0.5ms

[warkolm (Mark Walkom)]

Ok, but do you have Monitoring enabled?
What do your threadpools look like? CPU? heap?

[alexus]

I do, I think it's "basic" though, so I can't see threadpools, CPU or heap...

[warkolm (Mark Walkom)]

You can definitely see them in Kibana if you have it enabled.

If not, then the _cat/ APIs will be able to show you.

[alexus]

... sorry, I take it back, I was able to see that via node, but which node do I check? query running across multiple indices..

on one of the random data nodes: heap is about 20G, CPU utilization isn't showing..

indexing threads are ~0 and read threads are ~0

[warkolm (Mark Walkom)]

The node in your original post is koEOgkkDBhOVfKHA7QXBjj.

[alexus]

oh right, sorry)

I added another node and query didn't fail, so I guess you were right that it was under heavy load..

meanwhile, I'll do some reading of Thread Pool | Elasticsearch Reference [6.8] | Elastic

Thanks!