# Thread Pool Configuration - Max thread\_pool.size?

**URL:** <https://discuss.elastic.co/t/thread-pool-configuration-max-thread-pool-size/228089>\
**Category:** Elasticsearch\
**Created:** [April 15, 2020, 10:33am UTC](https://discuss.elastic.co/t/thread-pool-configuration-max-thread-pool-size/228089 "2020-04-15T10:33:57Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![Vipul\_Sharma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vipul_sharma/32/45741_2.png) [@Vipul\_Sharma](https://discuss.elastic.co/u/Vipul_Sharma)\
**Post date:** [April 15, 2020, 10:33am UTC](https://discuss.elastic.co/t/thread-pool-configuration-max-thread-pool-size/228089/1 "2020-04-15T10:33:57Z")

</div>

Hi All,

Kindly help me with my query

What could be the best possible configuration for a data-node running on a server having configuration

lscpu output\>

> Architecture: x86\_64  
> CPU op-mode(s): 32-bit, 64-bit  
> Byte Order: Little Endian  
> CPU(s): 28  
> On-line CPU(s) list: 0-27  
> Thread(s) per core: 1  
> Core(s) per socket: 2  
> Socket(s): 14  
> NUMA node(s): 1  
> Vendor ID: GenuineIntel  
> CPU family: 6  
> Model: 79  
> Model name: Intel(R) Xeon(R) CPU E5-2683 v4 @ 2.10GHz  
> Stepping: 1  
> CPU MHz: 2097.691  
> BogoMIPS: 4195.23  
> Hypervisor vendor: Xen  
> Virtualization type: full  
> L1d cache: 32K  
> L1i cache: 32K  
> L2 cache: 256K  
> L3 cache: 40960K  
> NUMA node0 CPU(s): 0-27

Current Settings are

```
thread_pool:
  write:
    size: 24
  search:
    size: 85
    queue_size: 500
    min_queue_size: 10
    max_queue_size: 1000
    auto_queue_frame_size: 2000
    target_response_time: 1s

```

and there are very high search queries on this server

```
node_name name active queue 
data-node-1 search 85 51 

```

because of which queue is getting filled up that is ok  
as our max\_queue\_size is 1000  
but es is rejecting the requests after the queue has reached 51 why so? is it because of target\_response\_time?  
and what could be a max value for thread\_pool.search.size?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [April 15, 2020, 10:44am UTC](https://discuss.elastic.co/t/thread-pool-configuration-max-thread-pool-size/228089/2 "2020-04-15T10:44:14Z")

</div>

Before giving any recommendations I have some questions:

- How many nodes do you have in the cluster?

- Which version of Elasticsearch are you using?

- What is the full output from the [cluster stats API](https://www.elastic.co/guide/en/elasticsearch/reference/7.6/cluster-stats.html)?

- How many concurrent queries are you expecing to need to support?

---

<div class="post-metadata">

**Author:** ![Vipul\_Sharma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vipul_sharma/32/45741_2.png) [@Vipul\_Sharma](https://discuss.elastic.co/u/Vipul_Sharma)\
**Post date:** [April 15, 2020, 11:05am UTC](https://discuss.elastic.co/t/thread-pool-configuration-max-thread-pool-size/228089/3 "2020-04-15T11:05:07Z")

</div>

We have 11 nodes in total

> 5 data nodes  
> 3 coordincating nodes  
> 3 master nodes

Version 6.5.4

GET /\_cluster/stats \>

> {  
> "\_nodes": {  
> "total": 11,  
> "successful": 11,  
> "failed": 0  
> },  
> "cluster\_name": "blackhole",  
> "cluster\_uuid": "GEbMwYZ1Q32OWJFzEdTQSA",  
> "timestamp": 1586948033165,  
> "status": "green",  
> "indices": {  
> "count": 110,  
> "shards": {  
> "total": 1004,  
> "primaries": 502,  
> "replication": 1,  
> "index": {  
> "shards": {  
> "min": 2,  
> "max": 10,  
> "avg": 9.127272727272727  
> },  
> "primaries": {  
> "min": 1,  
> "max": 5,  
> "avg": 4.5636363636363635  
> },  
> "replication": {  
> "min": 1,  
> "max": 1,  
> "avg": 1  
> }  
> }  
> },  
> "docs": {  
> "count": 367659960,  
> "deleted": 50731783  
> },  
> "store": {  
> "size\_in\_bytes": 1119225068695  
> },  
> "fielddata": {  
> "memory\_size\_in\_bytes": 23710584,  
> "evictions": 0  
> },  
> "query\_cache": {  
> "memory\_size\_in\_bytes": 4513623739,  
> "total\_count": 637231173997,  
> "hit\_count": 143510379985,  
> "miss\_count": 493720794012,  
> "cache\_size": 291378,  
> "cache\_count": 3239636308,  
> "evictions": 3239344930  
> },  
> "completion": {  
> "size\_in\_bytes": 0  
> },  
> "segments": {  
> "count": 11170,  
> "memory\_in\_bytes": 13853151308,  
> "terms\_memory\_in\_bytes": 13708166460,  
> "stored\_fields\_memory\_in\_bytes": 73873832,  
> "term\_vectors\_memory\_in\_bytes": 0,  
> "norms\_memory\_in\_bytes": 24145472,  
> "points\_memory\_in\_bytes": 18040064,  
> "doc\_values\_memory\_in\_bytes": 28925480,  
> "index\_writer\_memory\_in\_bytes": 0,  
> "version\_map\_memory\_in\_bytes": 0,  
> "fixed\_bit\_set\_memory\_in\_bytes": 164822672,  
> "max\_unsafe\_auto\_id\_timestamp": -1,  
> "file\_sizes": {}  
> }  
> },  
> "nodes": {  
> "count": {  
> "total": 11,  
> "data": 5,  
> "coordinating\_only": 3,  
> "master": 3,  
> "ingest": 0  
> },  
> "versions": [  
> "6.5.4"  
> ],  
> "os": {  
> "available\_processors": 200,  
> "allocated\_processors": 200,  
> "names": [  
> {  
> "name": "Linux",  
> "count": 11  
> }  
> ],  
> "mem": {  
> "total\_in\_bytes": 641698430976,  
> "free\_in\_bytes": 99411136512,  
> "used\_in\_bytes": 542287294464,  
> "free\_percent": 15,  
> "used\_percent": 85  
> }  
> },  
> "process": {  
> "cpu": {  
> "percent": 412  
> },  
> "open\_file\_descriptors": {  
> "min": 512,  
> "max": 3753,  
> "avg": 1741  
> }  
> },  
> "jvm": {  
> "max\_uptime\_in\_millis": 24200534975,  
> "versions": [  
> {  
> "version": "1.8.0\_201",  
> "vm\_name": "Java HotSpot(TM) 64-Bit Server VM",  
> "vm\_version": "25.201-b09",  
> "vm\_vendor": "Oracle Corporation",  
> "count": 11  
> }  
> ],  
> "mem": {  
> "heap\_used\_in\_bytes": 167080555032,  
> "heap\_max\_in\_bytes": 346575339520  
> },  
> "threads": 2920  
> },  
> "fs": {  
> "total\_in\_bytes": 3156332773376,  
> "free\_in\_bytes": 2000615645184,  
> "available\_in\_bytes": 2000615645184  
> },  
> "plugins": ,  
> "network\_types": {  
> "transport\_types": {  
> "security4": 11  
> },  
> "http\_types": {  
> "security4": 11  
> }  
> }  
> }  
> }

How many concurrent queries can our elastic cluster serve at it best?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [April 15, 2020, 11:19am UTC](https://discuss.elastic.co/t/thread-pool-configuration-max-thread-pool-size/228089/4 "2020-04-15T11:19:14Z")

</div>

It looks like you have a lot of small shards. How many of these does each query typically target?

---

<div class="post-metadata">

**Author:** ![Vipul\_Sharma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vipul_sharma/32/45741_2.png) [@Vipul\_Sharma](https://discuss.elastic.co/u/Vipul_Sharma)\
**Post date:** [April 15, 2020, 11:30am UTC](https://discuss.elastic.co/t/thread-pool-configuration-max-thread-pool-size/228089/5 "2020-04-15T11:30:56Z")

</div>

Each query target 5 shards at a time

Does number of Shards increase number of threads being used by a search query?  
is there any relation between number of threads and number of shards?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [April 15, 2020, 11:37am UTC](https://discuss.elastic.co/t/thread-pool-configuration-max-thread-pool-size/228089/6 "2020-04-15T11:37:07Z")

</div>

Each shard queried is put on the queue, so large number of concurrent queries or large number of shards queried can fill up the queues. Given the number of CPU cores you have a available and the size of the data, it is quite likely that your query performance is limited by disk I/O. Increasing queue sizes will not improve this, just allow you to queue more data for longer on the nodes.

What type of storage are you using?

---

<div class="post-metadata">

**Author:** ![Vipul\_Sharma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vipul_sharma/32/45741_2.png) [@Vipul\_Sharma](https://discuss.elastic.co/u/Vipul_Sharma)\
**Post date:** [April 15, 2020, 11:55am UTC](https://discuss.elastic.co/t/thread-pool-configuration-max-thread-pool-size/228089/7 "2020-04-15T11:55:40Z")

</div>

Storage is SSD

How many concurrent queries can our elastic cluster serve with the current server?

should we increase thread\_pool.search.size? to server more requests will that help?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [April 15, 2020, 12:13pm UTC](https://discuss.elastic.co/t/thread-pool-configuration-max-thread-pool-size/228089/8 "2020-04-15T12:13:18Z")

</div>

I do not know what type of queries or data you have and do not know what is currently limiting performance so it is hard to tell. I would not recommend altering the threadpool sizes unless you really know what you are doing as the defaults generally are quite good.

Even though you have local SSDs, look at disk I/O and iowait just to be sure this is not a bottleneck. Also check if you are seeing long or frequent GC. Then look at the load on the coordinating only nodes and check these are not limiting.

---

<div class="post-metadata">

**Author:** ![Vipul\_Sharma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vipul_sharma/32/45741_2.png) [@Vipul\_Sharma](https://discuss.elastic.co/u/Vipul_Sharma)\
**Post date:** [April 15, 2020, 12:22pm UTC](https://discuss.elastic.co/t/thread-pool-configuration-max-thread-pool-size/228089/9 "2020-04-15T12:22:58Z")

</div>

The issue only occurs when there is a sudden increase in the number of queries  
and i get this exception  
rejected execution of org.elasticsearch.common.util.concurrent.TimedRunnable@6fbf2782 on QueueResizingEsThreadPoolExecutor[name = data-node-3/search, queue capacity = 50, min queue capacity = 10, max queue capacity = 1000, frame size = 2000, targeted response rate = 1s, task execution EWMA = 255.6ms, adjustment amount = 50, org.elasticsearch.common.util.concurrent.QueueResizingEsThreadPoolExecutor@329b63d9[Running, pool size = 85, active threads = 85, queued tasks = 52, completed tasks = 4333428590]]

in this queued tasks are only 52 why so? it should go up to 1000?

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [April 15, 2020, 1:47pm UTC](https://discuss.elastic.co/t/thread-pool-configuration-max-thread-pool-size/228089/10 "2020-04-15T13:47:28Z")

</div>

> [@Vipul\_Sharma](#):
>
> in this queued tasks are only 52 why so? it should go up to 1000?

You should remove your adjustments to the default thread pool configuration. The default behaviour is the recommended one and will get the best out of your cluster. If you are having performance problems (e.g. rejections) then the solution is not to adjust the thread pool configuration.

---

<div class="post-metadata">

**Author:** ![Vipul\_Sharma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vipul_sharma/32/45741_2.png) [@Vipul\_Sharma](https://discuss.elastic.co/u/Vipul_Sharma)\
**Post date:** [April 16, 2020, 4:32am UTC](https://discuss.elastic.co/t/thread-pool-configuration-max-thread-pool-size/228089/11 "2020-04-16T04:32:52Z")

</div>

Thank you for all the information. This will be really helpful 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 14, 2020, 4:32am UTC](https://discuss.elastic.co/t/thread-pool-configuration-max-thread-pool-size/228089/12 "2020-05-14T04:32:54Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.