# Elasticsearch has accumulated a lot of pending tasks, and stop indexing **URL:** \ **Category:** Elasticsearch\ **Created:** [August 23, 2023, 12:57am UTC](https://discuss.elastic.co/t/elasticsearch-has-accumulated-a-lot-of-pending-tasks-and-stop-indexing/341414 "2023-08-23T00:57:21Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![ShanYang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shanyang/32/124863_2.png) [@ShanYang](https://discuss.elastic.co/u/ShanYang)\ **Post date:** [August 23, 2023, 12:57am UTC](https://discuss.elastic.co/t/elasticsearch-has-accumulated-a-lot-of-pending-tasks-and-stop-indexing/341414/1 "2023-08-23T00:57:21Z")
Phenomenon: When generating a new index across days, elasticsearch stop indexing. The cluster health show green but with many pending task. Used Get \_tasks?, I saw thousands transport task and hundreds direct task. Cluster recover after killing master node. It seems that the master node is stuck. Everything returned to normal when the master node was changed. Is any suggestion about this? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [August 23, 2023, 5:42am UTC](https://discuss.elastic.co/t/elasticsearch-has-accumulated-a-lot-of-pending-tasks-and-stop-indexing/341414/2 "2023-08-23T05:42:20Z")
Sounds like a bug. What version are you running? ---
**Author:** ![ShanYang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shanyang/32/124863_2.png) [@ShanYang](https://discuss.elastic.co/u/ShanYang)\ **Post date:** [August 23, 2023, 5:58am UTC](https://discuss.elastic.co/t/elasticsearch-has-accumulated-a-lot-of-pending-tasks-and-stop-indexing/341414/3 "2023-08-23T05:58:23Z")
version: 7.17.2 ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [August 23, 2023, 5:59am UTC](https://discuss.elastic.co/t/elasticsearch-has-accumulated-a-lot-of-pending-tasks-and-stop-indexing/341414/4 "2023-08-23T05:59:17Z")
Can you share the output of `GET _tasks?detailed` and `GET _cluster/pending_tasks` from the time of the problem? ---
**Author:** ![ShanYang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shanyang/32/124863_2.png) [@ShanYang](https://discuss.elastic.co/u/ShanYang)\ **Post date:** [August 23, 2023, 7:50am UTC](https://discuss.elastic.co/t/elasticsearch-has-accumulated-a-lot-of-pending-tasks-and-stop-indexing/341414/5 "2023-08-23T07:50:38Z")
I'm not sure if providing only the first few tasks is enough. From my perspective, there are plenty of tasks that look like this one. As for the pending tasks, I forgot to note them down. Hope this can help. { "nodes" : { "XXX" : { "name" : "es-cluster-tier1-2", "transport\_address" : "172.24.155.178:9300", "host" : "172.24.155.178", "ip" : "172.24.155.178:9300", "roles" : [ "data\_content", "data\_hot", "ingest", "master", "ml", "remote\_cluster\_client", "transform" ], "attributes" : { "ml.machine\_memory" : "8589934592", "ml.max\_open\_jobs" : "512", "xpack.installed" : "true", "ml.max\_jvm\_size" : "4294967296", "transform.node" : "true" }, "tasks" : { "XXX:110229137" : { "node" : "XXX", "id" : 110229137, "type" : "transport", "action" : "indices:data/write/bulk", "start\_time\_in\_millis" : 1692748957757, "running\_time\_in\_nanos" : 1391395119423, "cancellable" : false, "headers" : { } }, "XXX:110229136" : { "node" : "XXX", "id" : 110229136, "type" : "transport", "action" : "indices:data/write/bulk", "start\_time\_in\_millis" : 1692748957704, "running\_time\_in\_nanos" : 1391447831580, "cancellable" : false, "headers" : { } }, "XXX:110220947" : { "node" : "XXX", "id" : 110220947, "type" : "transport", "action" : "indices:data/write/bulk[s]", "start\_time\_in\_millis" : 1692748860676, "running\_time\_in\_nanos" : 1488476466847, "cancellable" : false, "parent\_task\_id" : "XXX:110220946", "headers" : { } }, ... ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [August 23, 2023, 9:14am UTC](https://discuss.elastic.co/t/elasticsearch-has-accumulated-a-lot-of-pending-tasks-and-stop-indexing/341414/6 "2023-08-23T09:14:30Z")
> [@ShanYang](#): > > I'm not sure if providing only the first few tasks is enough. Not really, no, we'd need to see the full output of both APIs. Without the pending tasks I don't think we can answer your questions. Please capture that if it happens again. ---
**Author:** ![ShanYang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shanyang/32/124863_2.png) [@ShanYang](https://discuss.elastic.co/u/ShanYang)\ **Post date:** [August 25, 2023, 9:25am UTC](https://discuss.elastic.co/t/elasticsearch-has-accumulated-a-lot-of-pending-tasks-and-stop-indexing/341414/7 "2023-08-25T09:25:34Z")
ok, I will keep watch because it is unpredictable when this phenomenon will occur. ---
**Author:** ![ShanYang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shanyang/32/124863_2.png) [@ShanYang](https://discuss.elastic.co/u/ShanYang)\ **Post date:** [September 5, 2023, 6:07am UTC](https://discuss.elastic.co/t/elasticsearch-has-accumulated-a-lot-of-pending-tasks-and-stop-indexing/341414/8 "2023-09-05T06:07:18Z")
it happend again. The pending task are as belows: insertOrder timeInQueue priority source ``` 42855 55.2m NORMAL cluster_reroute(reroute after starting shards) 42880 55.1m NORMAL ilm-execute-cluster-state-steps [{"phase":"new","action":"init","name":"init"} => {"phase":"new","action":"complete","name":"complete"}] 42909 55.1m NORMAL ilm-execute-cluster-state-steps [{"phase":"new","action":"init","name":"init"} => {"phase":"new","action":"complete","name":"complete"}] 42940 55.1m NORMAL ilm-execute-cluster-state-steps [{"phase":"new","action":"init","name":"init"} => {"phase":"new","action":"complete","name":"complete"}] ``` the following tasks are similar to ilm-execute-cluster-state-steps task. So it seems cause by cluster\_reroute? Get tasks action task\_id parent\_task\_id type start\_time timestamp running\_time ``` indices:data/write/bulk _Yn1U8jOSvWhgfaEzIVlIg:79310960 - transport 1.69388E+12 02:31:12 50.4m indices:data/write/bulk jKFbZ3SNSBWFI6JWUv5L7g:85644667 - transport 1.69388E+12 02:31:14 50.3m indices:data/write/bulk _Yn1U8jOSvWhgfaEzIVlIg:79311961 - transport 1.69388E+12 02:31:16 50.3m indices:data/write/bulk _Yn1U8jOSvWhgfaEzIVlIg:79312451 - transport 1.69388E+12 02:31:18 50.3m ``` task detail "\_Yn1U8jOSvWhgfaEzIVlIg:79310960" : { "node" : "\_Yn1U8jOSvWhgfaEzIVlIg", "id" : 79310960, "type" : "transport", "action" : "indices:data/write/bulk", "start\_time\_in\_millis" : 1693881072568, "running\_time\_in\_nanos" : 3524128366857, "cancellable" : false, "headers" : { } }, "jKFbZ3SNSBWFI6JWUv5L7g:85644667" : { "node" : "jKFbZ3SNSBWFI6JWUv5L7g", "id" : 85644667, "type" : "transport", "action" : "indices:data/write/bulk", "start\_time\_in\_millis" : 1693881074669, "running\_time\_in\_nanos" : 3522028629897, "cancellable" : false, "headers" : { } }, ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [September 5, 2023, 6:40am UTC](https://discuss.elastic.co/t/elasticsearch-has-accumulated-a-lot-of-pending-tasks-and-stop-indexing/341414/9 "2023-09-05T06:40:04Z")
Can you share the _full_ output of `GET _tasks?detailed` and `GET _cluster/pending_tasks` from the time of the problem? ---
**Author:** ![ShanYang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shanyang/32/124863_2.png) [@ShanYang](https://discuss.elastic.co/u/ShanYang)\ **Post date:** [September 5, 2023, 6:45am UTC](https://discuss.elastic.co/t/elasticsearch-has-accumulated-a-lot-of-pending-tasks-and-stop-indexing/341414/10 "2023-09-05T06:45:14Z")
I sorted the pending task by insertOrder, so the following task i think is not related. (and the rest tasks are all same with "ilm-execute-cluster-state-steps [{"phase":"new","action":"init","name":"init"} =\> {"phase":"new","action":"complete","name":"complete"}]", except insertOrder/timeInQueue. And I add tasks?detailed at previous reply. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [September 5, 2023, 6:55am UTC](https://discuss.elastic.co/t/elasticsearch-has-accumulated-a-lot-of-pending-tasks-and-stop-indexing/341414/11 "2023-09-05T06:55:20Z")
Sorry, you need to share the complete output to the APIs I indicated. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [September 6, 2023, 7:35am UTC](https://discuss.elastic.co/t/elasticsearch-has-accumulated-a-lot-of-pending-tasks-and-stop-indexing/341414/14 "2023-09-06T07:35:48Z")
You're still not using `GET _cluster/pending_tasks` - the output you have shared is from `GET _cat/pending_tasks`. The difference is important. I think it would also help to see the output from `GET _nodes/_master/hot_threads?threads=9999` while the cluster is in the problematic state. ---
**Author:** ![ShanYang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shanyang/32/124863_2.png) [@ShanYang](https://discuss.elastic.co/u/ShanYang)\ **Post date:** [September 6, 2023, 9:16am UTC](https://discuss.elastic.co/t/elasticsearch-has-accumulated-a-lot-of-pending-tasks-and-stop-indexing/341414/15 "2023-09-06T09:16:53Z")
I was wrong, sorry. It seems like I'll have to collect it again next time. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 4, 2023, 9:17am UTC](https://discuss.elastic.co/t/elasticsearch-has-accumulated-a-lot-of-pending-tasks-and-stop-indexing/341414/16 "2023-10-04T09:17:28Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.