# Elastic Cluster Rejected Execution Exception

**URL:** <https://discuss.elastic.co/t/elastic-cluster-rejected-execution-exception/219143>\
**Category:** Elasticsearch\
**Created:** [February 13, 2020, 7:37am UTC](https://discuss.elastic.co/t/elastic-cluster-rejected-execution-exception/219143 "2020-02-13T07:37:56Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Garima\_Tripathi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/garima_tripathi/32/46325_2.png) [@Garima\_Tripathi](https://discuss.elastic.co/u/Garima_Tripathi)\
**Post date:** [February 13, 2020, 7:37am UTC](https://discuss.elastic.co/t/elastic-cluster-rejected-execution-exception/219143/1 "2020-02-13T07:37:56Z")

</div>

Hi,

I have the following cluster configuration :

Master -3 node (4 cpu, 20 gb ram)  
coordinating -3 node (16 cpu, 98 GB,)  
data- 5 node (28 cpu, 48 RAM)

And thread pool configuration is following  
thread\_pool:

search:  
size: 64  
queue\_size: 500  
min\_queue\_size: 10  
max\_queue\_size: 1000  
auto\_queue\_frame\_size: 2000  
target\_response\_time: 1s  
http.enabled: true

We have some complex queries and there are regular updates on around 25% of data on the cluster.

I have observed that after the concurrency of approx 19k rpm, the cluster starts to reject the requests. I expect the traffic to increase further.

Please tell me what configuration changes can help me so that no requests are rejected with this cluster configuration.

Thanks.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [February 13, 2020, 7:47am UTC](https://discuss.elastic.co/t/elastic-cluster-rejected-execution-exception/219143/2 "2020-02-13T07:47:44Z")

</div>

Have you identified what is limiting performance? Is it disk I/O on the data nodes? Long or frequent GC on data or coordinating nodes? Is CPU saturated on some nodes? Is networking proving to be a bottleneck?

How much data do you have in the cluster? How many indices and shards? What type of complex queries are you running? How do you perform these regular updates?

---

<div class="post-metadata">

**Author:** ![Garima\_Tripathi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/garima_tripathi/32/46325_2.png) [@Garima\_Tripathi](https://discuss.elastic.co/u/Garima_Tripathi)\
**Post date:** [February 13, 2020, 10:43am UTC](https://discuss.elastic.co/t/elastic-cluster-rejected-execution-exception/219143/3 "2020-02-13T10:43:28Z")

</div>

Hi I am attaching the disk I/O stats as well as GC Logs Below:

[Co-ordinating node](https://drive.google.com/open?id=1rO7Qigx29zTVpipqK_jAGDoftDPco6e3)[Data Node](https://drive.google.com/open?id=11g2NTtvplA3IJZDw8XjSg-dQjDgMO8Ol)

Networking is not a bottleneck as what i found.

I have around 120GB of data with 6 indices out of which 1 is major with around 110GB of data which is queried frequently. All indices have 5 shards. Partial Updates are done throughout the day on the largest index.

CPU reaches upto 85% when load is high.

Sample Complex Query :  
[Sample Query](https://drive.google.com/open?id=1xer7HttkRZ4O0lxQ4w4dsTh-sWxo67KKvZna0cGnxiU)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 12, 2020, 10:43am UTC](https://discuss.elastic.co/t/elastic-cluster-rejected-execution-exception/219143/4 "2020-03-12T10:43:29Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.