# Rejected execution (queue capacity 1000)

**URL:** <https://discuss.elastic.co/t/rejected-execution-queue-capacity-1000/89954>\
**Category:** Elasticsearch\
**Created:** [June 19, 2017, 3:14pm UTC](https://discuss.elastic.co/t/rejected-execution-queue-capacity-1000/89954 "2017-06-19T15:14:21Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![elbori76](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/elbori76/32/14748_2.png) [@elbori76](https://discuss.elastic.co/u/elbori76)\
**Post date:** [June 19, 2017, 3:14pm UTC](https://discuss.elastic.co/t/rejected-execution-queue-capacity-1000/89954/1 "2017-06-19T15:14:21Z")

</div>

Receiving the following error in elasticsearch cluster logs  
Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution (queue capacity 1000) on org.elasticsearch.transport.netty.MessageChannelHandler$RequestHandler

Was able to notice the following during one of the times a search query was hung  
[http://node01.testing.local:9200/\_nodes/node01/stats/thread\_pool?human&pretty](http://node01.testing.local:9200/_nodes/node01/stats/thread_pool?human&pretty)  
"search": {  
"threads": 24,  
"queue": 1000,  
"active": 24,  
"rejected": 300,  
"largest": 24,  
"completed": 141910

Cluster Information  
We have a 11 node ELK cluster in which 3 master nodes, 5 data nodes and 3 logstash nodes.

Data nodes are configured as followed

8vCPU  
64GB RAM (32 allocated to ES\_HEAP\_SIZE)  
1.4TB iSCSI Volume with 700GBs used  
Swap is disabled.

---

<div class="post-metadata">

**Author:** ![polyfractal](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/polyfractal/32/48162_2.png) [@polyfractal](https://discuss.elastic.co/u/polyfractal)\
**Post date:** [June 20, 2017, 9:32pm UTC](https://discuss.elastic.co/t/rejected-execution-queue-capacity-1000/89954/2 "2017-06-20T21:32:55Z")

</div>

Well, it basically means that you've got 1000 search requests that have queued up waiting to run, and once the limit is reached ES just starts aborting new requests.

So you'll need to figure out the bottleneck. Some options:

- Your clients are simply sending too many queries too quickly in a fast burst, overwhelming the queue. You can monitor this with Node Stats over time to see if it's bursty or smooth
- You've got some very slow queries which get "stuck" for a long time, eating up threads and causing the queue to back up. You can enable the slow log to see if there are queries that are taking an exceptionally long time, then try to tune those
- There may potentially be "unending" scripts written in Groovy or something. E.g. a loop that never exits, causing the thread to spin forever.
- Your hardware may be under-provisioned for your workload, and bottlenecking on some resource (disk, cpu, etc)
- A temporary hiccup from your iSCSI target, which causes all the in-flight operations to block waiting for the disks to come back. It wouldn't take a big latency hiccup to seriously backup a busy cluster... ES generally expects disks to always be available.
- Heavy garbage collections could cause problems too. Check Node Stats to see if there are many/long old gen GCs running

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 18, 2017, 9:33pm UTC](https://discuss.elastic.co/t/rejected-execution-queue-capacity-1000/89954/3 "2017-07-18T21:33:07Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.