Elasticsearch has accumulated a lot of pending tasks, and stop indexing

Elastic Stack Elasticsearch
8

it happend again.
The pending task are as belows:

insertOrder timeInQueue priority source

  42855       55.2m NORMAL   cluster_reroute(reroute after starting shards)
  42880       55.1m NORMAL   ilm-execute-cluster-state-steps [{"phase":"new","action":"init","name":"init"} => {"phase":"new","action":"complete","name":"complete"}]
  42909       55.1m NORMAL   ilm-execute-cluster-state-steps [{"phase":"new","action":"init","name":"init"} => {"phase":"new","action":"complete","name":"complete"}]
  42940       55.1m NORMAL   ilm-execute-cluster-state-steps [{"phase":"new","action":"init","name":"init"} => {"phase":"new","action":"complete","name":"complete"}]

the following tasks are similar to ilm-execute-cluster-state-steps task.

So it seems cause by cluster_reroute?

Get tasks
action task_id parent_task_id type start_time timestamp running_time

  indices:data/write/bulk	_Yn1U8jOSvWhgfaEzIVlIg:79310960	-	transport	1.69388E+12	02:31:12	50.4m

  indices:data/write/bulk	jKFbZ3SNSBWFI6JWUv5L7g:85644667	-	transport	1.69388E+12	02:31:14	50.3m

  indices:data/write/bulk	_Yn1U8jOSvWhgfaEzIVlIg:79311961	-	transport	1.69388E+12	02:31:16	50.3m

  indices:data/write/bulk	_Yn1U8jOSvWhgfaEzIVlIg:79312451	-	transport	1.69388E+12	02:31:18	50.3m

task detail
"_Yn1U8jOSvWhgfaEzIVlIg:79310960" : {
"node" : "_Yn1U8jOSvWhgfaEzIVlIg",
"id" : 79310960,
"type" : "transport",
"action" : "indices:data/write/bulk",
"start_time_in_millis" : 1693881072568,
"running_time_in_nanos" : 3524128366857,
"cancellable" : false,
"headers" : { }
},
"jKFbZ3SNSBWFI6JWUv5L7g:85644667" : {
"node" : "jKFbZ3SNSBWFI6JWUv5L7g",
"id" : 85644667,
"type" : "transport",
"action" : "indices:data/write/bulk",
"start_time_in_millis" : 1693881074669,
"running_time_in_nanos" : 3522028629897,
"cancellable" : false,
"headers" : { }
},