# Thread pool size and the cluster settings API

**URL:** <https://discuss.elastic.co/t/thread-pool-size-and-the-cluster-settings-api/249741>\
**Category:** Elasticsearch\
**Created:** [September 24, 2020, 6:19am UTC](https://discuss.elastic.co/t/thread-pool-size-and-the-cluster-settings-api/249741 "2020-09-24T06:19:19Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Pradyumna\_Achar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pradyumna_achar/32/99780_2.png) [@Pradyumna\_Achar](https://discuss.elastic.co/u/Pradyumna_Achar)\
**Post date:** [September 24, 2020, 6:19am UTC](https://discuss.elastic.co/t/thread-pool-size-and-the-cluster-settings-api/249741/1 "2020-09-24T06:19:19Z")

</div>

Hello,

I am a bit confused on the cluster settings API. The thread pool settings seem to be specific to a node.

For example, in the same cluster, some nodes have 1 CPU and some have 5 CPUs. When I run the `_cluster/settings` API, the settings shown are different based on where I run them:

```auto
curl -s '10.0.58.0:9200/_cluster/settings?include_defaults' | jq '.defaults.thread_pool.write'
{
  "queue_size": "200",
  "size": "5"
}

```

```auto
curl -s '10.0.33.29:9200/_cluster/settings?include_defaults' | jq '.defaults.thread_pool.write'
{
  "queue_size": "200",
  "size": "1"
}

```

(The ones with the 5 CPU are the data nodes, BTW)

If I wanted to update this setting and increase the `queue_size` to say 300 or `size` to 8, should I run the `-X PUT` version separately on each of the data nodes where I want it increased?

I am a bit confused because I was under the impression that the `_cluster/settings` API is for working with cluster-wide settings as the page says "Use this API to review and change cluster-wide settings.", and I am not sure if I am misunderstanding anything here with respect to these different queue size responses based on which node responds to the API. [https://www.elastic.co/guide/en/elasticsearch/reference/7.0/cluster-update-settings.html](https://www.elastic.co/guide/en/elasticsearch/reference/7.0/cluster-update-settings.html)

Thank you  
Pradyumna

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 24, 2020, 6:38am UTC](https://discuss.elastic.co/t/thread-pool-size-and-the-cluster-settings-api/249741/2 "2020-09-24T06:38:39Z")

</div>

I would suggest reading [Should I increase my threadpool size if I get rejected executions or HTTP 429 responses?](https://discuss.elastic.co/t/should-i-increase-my-threadpool-size-if-i-get-rejected-executions-or-http-429-responses/241044) before changing threadpool settings.

---

<div class="post-metadata">

**Author:** ![Pradyumna\_Achar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pradyumna_achar/32/99780_2.png) [@Pradyumna\_Achar](https://discuss.elastic.co/u/Pradyumna_Achar)\
**Post date:** [September 24, 2020, 7:21am UTC](https://discuss.elastic.co/t/thread-pool-size-and-the-cluster-settings-api/249741/3 "2020-09-24T07:21:19Z")

</div>

Yes, I get that; the queue is not full all the time; but occasionally it does reject a few tens of requests per thousand.  
But when these tens of requests get rejected, the system that sends data to Elasticsearch takes a costly route to perform a retry, which quickly magnifies upstream and takes several seconds to resolve.

If the queue was just a bit larger to accommodate these 30-40 extra requests once in a while, or if there were more threads consuming from the pool, it would solve the problem as the CPU utilization of Elasticsearch nodes is well under limits.

Hence the interest in this `_cluster/settings` API for the pool sizes

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [September 24, 2020, 8:08am UTC](https://discuss.elastic.co/t/thread-pool-size-and-the-cluster-settings-api/249741/4 "2020-09-24T08:08:01Z")

</div>

Elasticsearch is more often limited by disk performance than CPU so it may be worth monitoring this at peak times. What is the specification of your cluster? What type of hardware are you using?

---

<div class="post-metadata">

**Author:** ![Pradyumna\_Achar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pradyumna_achar/32/99780_2.png) [@Pradyumna\_Achar](https://discuss.elastic.co/u/Pradyumna_Achar)\
**Post date:** [September 24, 2020, 8:32am UTC](https://discuss.elastic.co/t/thread-pool-size-and-the-cluster-settings-api/249741/5 "2020-09-24T08:32:37Z")

</div>

The load is at a fairly constant rate of homogeneous requests from machine-generated data, so there's no peak in the pattern. But yes, the disk might be misbehaving once in a while; It's an AWS EBS gp2 volume and they say on their web page that "AWS designs gp2 volumes to deliver their provisioned performance 99% of the time." So, for that remaining 1% of the time, the queue could very well fill up when operating at near the provisioned iops threshold of the volume.

While one solution would be to have a disk with a higher iops, it is more expensive than increasing the queue size by a few tens.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 22, 2020, 8:32am UTC](https://discuss.elastic.co/t/thread-pool-size-and-the-cluster-settings-api/249741/6 "2020-10-22T08:32:57Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.