Rejected execution

Elastic Stack Elasticsearch
1

Hello World!

I'm using Elasticsearch 6.8.10 and trying to run a query, yet running into following issue:

{
   "shard" : 3,
   "node" : "koEOgkkDBhOVfKHA7QXBjj",
   "reason" : {
      "reason" : "rejected execution of org.elasticsearch.common.util.concurrent.TimedRunnable@26c03d4a on QueueResizingEsThreadPoolExecutor[name = esdX/search, queue capacity = 1000, min queue capacity = 1000, max queue capacity = 1000, frame size = 2000, targeted response rate = 1s, task execution EWMA = 968.1ms, adjustment amount = 50, org.elasticsearch.common.util.concurrent.QueueResizingEsThreadPoolExecutor@70be0765[Running, pool size = 13, active threads = 13, queued tasks = 1000, completed tasks = 616499351]]",
      "type" : "es_rejected_execution_exception"
   },
   "index" : "X-X-X"
}

Please advise.
Thanks in advance!

2

Sounds like Elasticsearch is overloaded.
Do you have Monitoring enabled?

3
  • search rate = ~2-3k
  • search latency (ms) = ~1ms
  • index rate = ~3-4k
  • indexing latency (ms) = ~>0.5ms
4

Ok, but do you have Monitoring enabled?
What do your threadpools look like? CPU? heap?

5

I do, I think it's "basic" though, so I can't see threadpools, CPU or heap...

6

You can definitely see them in Kibana if you have it enabled.

If not, then the _cat/ APIs will be able to show you.

7

... sorry, I take it back, I was able to see that via node, but which node do I check? query running across multiple indices..

on one of the random data nodes: heap is about 20G, CPU utilization isn't showing..

indexing threads are ~0 and read threads are ~0

8

The node in your original post is koEOgkkDBhOVfKHA7QXBjj.

9

oh right, sorry)

I added another node and query didn't fail, so I guess you were right that it was under heavy load..

meanwhile, I'll do some reading of Thread Pool | Elasticsearch Reference [6.8] | Elastic

Thanks!

1 Like
10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.