# Running queue tasks more than max queue capacity

**URL:** <https://discuss.elastic.co/t/running-queue-tasks-more-than-max-queue-capacity/316496>\
**Category:** Elasticsearch\
**Created:** [October 13, 2022, 7:15am UTC](https://discuss.elastic.co/t/running-queue-tasks-more-than-max-queue-capacity/316496 "2022-10-13T07:15:20Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![liuhuang9492](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/liuhuang9492/32/105622_2.png) [@liuhuang9492](https://discuss.elastic.co/u/liuhuang9492)\
**Post date:** [October 13, 2022, 7:15am UTC](https://discuss.elastic.co/t/running-queue-tasks-more-than-max-queue-capacity/316496/1 "2022-10-13T07:15:20Z")

</div>

version 6.3.2, elasticsearch, version 6.3.2, has 5 master nodes and 15 data nodes.  
The following problems occur in the log:

```auto
Caused by: org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: 
rejected execution of org.elasticsearch.common.util.concurrent.TimedRunnable@1d9f2200 on QueueResizingEsThreadPoolExecutor
[name = xxxxx-data/search, queue capacity = 1000, min queue capacity = 1000, max queue capacity = 1000, frame size = 2000, targeted response rate = 1s, task execution EWMA = 284.8micros, adjustment amount = 50, org.elasticsearch.common.util.concurrent.QueueResizingEsThreadPoolExecutor@1f60aa97
[Running, pool size = 49, active threads = 49, queued tasks = 56999, completed tasks = 336035534]]

```

the running queue tasks is more than max queue capacity.  
In this case, the`_cat/nodes`operation times out:  
`collector [node_stats] timed out when collecting data`

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [October 13, 2022, 10:57pm UTC](https://discuss.elastic.co/t/running-queue-tasks-more-than-max-queue-capacity/316496/2 "2022-10-13T22:57:11Z")

</div>

Elasticsearch 6.X is [EOL](https://www.elastic.co/support/eol), please upgrade as a matter of urgency.

Basically this is saying your cluster is overloaded. What is the output from the [`_cluster/stats?pretty&human` API](https://www.elastic.co/guide/en/elasticsearch/reference/8.4/cluster-stats.html)?

---

<div class="post-metadata">

**Author:** ![liuhuang9492](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/liuhuang9492/32/105622_2.png) [@liuhuang9492](https://discuss.elastic.co/u/liuhuang9492)\
**Post date:** [October 14, 2022, 1:50am UTC](https://discuss.elastic.co/t/running-queue-tasks-more-than-max-queue-capacity/316496/3 "2022-10-14T01:50:00Z")

</div>

> [@liuhuang9492](#):
>
> `collector [node_stats] timed out when collecting data`

sorry, the cluster is green health now. and I do not keep the `_cluster/stats` information.

but when I run `_cluster/health?pretty` on master node when the cluster is problem, it show green health. and I run any rest api on data node ,it show timeout.

Even if the front-end traffic is limited to very low levels, the cluster logs still throw timeout and queue exceptions.Finally, all indexes are closed first, and then open, it is found that the log does not throw this error, and gradually open the front-end traffic limit, the cluster is still normal.

There are two people who want to confirm:

1. Check whether queue tasks in running are true requests. After restarting the cluster, there are still a large number of queue exceptions in this log.

2. Will the queue be cleared by the close index?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [October 14, 2022, 3:30am UTC](https://discuss.elastic.co/t/running-queue-tasks-more-than-max-queue-capacity/316496/4 "2022-10-14T03:30:23Z")

</div>

> [@liuhuang9492](#):
>
> I do not keep the `_cluster/stats` information.

It's stored in the cluster, you can query at any time.

---

<div class="post-metadata">

**Author:** ![liuhuang9492](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/liuhuang9492/32/105622_2.png) [@liuhuang9492](https://discuss.elastic.co/u/liuhuang9492)\
**Post date:** [October 14, 2022, 6:11am UTC](https://discuss.elastic.co/t/running-queue-tasks-more-than-max-queue-capacity/316496/5 "2022-10-14T06:11:35Z")

</div>

{  
"\_nodes" : {  
"total" : 20,  
"successful" : 20,  
"failed" : 0  
},  
"cluster\_name" : "cluster\_name2",  
"timestamp" : 1665727685194,  
"status" : "green",  
"indices" : {  
"count" : 99,  
"shards" : {  
"total" : 632,  
"primaries" : 281,  
"replication" : 1.2491103202846976,  
"index" : {  
"shards" : {  
"min" : 2,  
"max" : 40,  
"avg" : 6.383838383838384  
},  
"primaries" : {  
"min" : 1,  
"max" : 10,  
"avg" : 2.8383838383838382  
},  
"replication" : {  
"min" : 1.0,  
"max" : 3.0,  
"avg" : 1.2626262626262625  
}  
}  
},  
"docs" : {  
"count" : 2038649028,  
"deleted" : 651890076  
},  
"store" : {  
"size" : "1.5tb",  
"size\_in\_bytes" : 1718697492594  
},  
"fielddata" : {  
"memory\_size" : "60.6kb",  
"memory\_size\_in\_bytes" : 62120,  
"evictions" : 0  
},  
"query\_cache" : {  
"memory\_size" : "0b",  
"memory\_size\_in\_bytes" : 0,  
"total\_count" : 0,  
"hit\_count" : 0,  
"miss\_count" : 0,  
"cache\_size" : 0,  
"cache\_count" : 0,  
"evictions" : 0  
},  
"completion" : {  
"size" : "0b",  
"size\_in\_bytes" : 0  
},  
"segments" : {  
"count" : 6392,  
"memory" : "5.7gb",  
"memory\_in\_bytes" : 6166553349,  
"terms\_memory" : "5.3gb",  
"terms\_memory\_in\_bytes" : 5782436211,  
"stored\_fields\_memory" : "276.3mb",  
"stored\_fields\_memory\_in\_bytes" : 289747000,  
"term\_vectors\_memory" : "0b",  
"term\_vectors\_memory\_in\_bytes" : 0,  
"norms\_memory" : "2mb",  
"norms\_memory\_in\_bytes" : 2189888,  
"points\_memory" : "75.9mb",  
"points\_memory\_in\_bytes" : 79617346,  
"doc\_values\_memory" : "11.9mb",  
"doc\_values\_memory\_in\_bytes" : 12562904,  
"index\_writer\_memory" : "4.3mb",  
"index\_writer\_memory\_in\_bytes" : 4535992,  
"version\_map\_memory" : "0b",  
"version\_map\_memory\_in\_bytes" : 0,  
"fixed\_bit\_set" : "287.6kb",  
"fixed\_bit\_set\_memory\_in\_bytes" : 294576,  
"max\_unsafe\_auto\_id\_timestamp" : 1665705603283,  
"file\_sizes" : { }  
}  
},  
"nodes" : {  
"count" : {  
"total" : 20,  
"data" : 15,  
"coordinating\_only" : 0,  
"master" : 5,  
"ingest" : 20  
},  
"versions" : [  
"6.3.2"  
],  
"os" : {  
"available\_processors" : 640,  
"allocated\_processors" : 640,  
"names" : [  
{  
"name" : "Linux",  
"count" : 20  
}  
],  
"mem" : {  
"total" : "3.6tb",  
"total\_in\_bytes" : 4028666773504,  
"free" : "370.8gb",  
"free\_in\_bytes" : 398206930944,  
"used" : "3.3tb",  
"used\_in\_bytes" : 3630459842560,  
"free\_percent" : 10,  
"used\_percent" : 90  
}  
},  
"process" : {  
"cpu" : {  
"percent" : 143  
},  
"open\_file\_descriptors" : {  
"min" : 1260,  
"max" : 1815,  
"avg" : 1520  
}  
},  
"jvm" : {  
"max\_uptime" : "11.6d",  
"max\_uptime\_in\_millis" : 1005554496,  
"versions" : [  
{  
"version" : "1.8.0\_192",  
"vm\_name" : "Java HotSpot(TM) 64-Bit Server VM",  
"vm\_version" : "25.192-b12",  
"vm\_vendor" : "Oracle Corporation",  
"count" : 20  
}  
],  
"mem" : {  
"heap\_used" : "272.4gb",  
"heap\_used\_in\_bytes" : 292565225120,  
"heap\_max" : "616.2gb",  
"heap\_max\_in\_bytes" : 661707816960  
},  
"threads" : 6562  
},  
"fs" : {  
"total" : "54.5tb",  
"total\_in\_bytes" : 59982473994240,  
"free" : "46.6tb",  
"free\_in\_bytes" : 51261142790144,  
"available" : "46.6tb",  
"available\_in\_bytes" : 51261142790144  
},  
"plugins" : [  
{  
"name" : "analysis-jieba",  
"version" : "6.3.2",  
"elasticsearch\_version" : "6.3.2",  
"java\_version" : "1.8",  
"description" : "A jieba analysis of plugins for Elasticsearch",  
"classname" : "org.elasticsearch.plugin.analysis.jieba.AnalysisJiebaPlugin",  
"extended\_plugins" : ,  
"has\_native\_controller" : false  
},  
{  
"name" : "analysis-ik",  
"version" : "6.3.2",  
"elasticsearch\_version" : "6.3.2",  
"java\_version" : "1.8",  
"description" : "IK Analyzer for Elasticsearch",  
"classname" : "org.elasticsearch.plugin.analysis.ik.AnalysisIkPlugin",  
"extended\_plugins" : ,  
"has\_native\_controller" : false  
}  
],  
"network\_types" : {  
"transport\_types" : {  
"security4" : 20  
},  
"http\_types" : {  
"security4" : 20  
}  
}  
}  
}

---

<div class="post-metadata">

**Author:** ![liuhuang9492](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/liuhuang9492/32/105622_2.png) [@liuhuang9492](https://discuss.elastic.co/u/liuhuang9492)\
**Post date:** [October 14, 2022, 6:13am UTC](https://discuss.elastic.co/t/running-queue-tasks-more-than-max-queue-capacity/316496/6 "2022-10-14T06:13:35Z")

</div>

The preceding information is about the `_cluster/stats?pretty&human` . and I had add 5 data nodes when cluster throw the `timeout` error.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 11, 2022, 6:13am UTC](https://discuss.elastic.co/t/running-queue-tasks-more-than-max-queue-capacity/316496/7 "2022-11-11T06:13:49Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.