# Pending\_tasks has millions of entries, many of with are exact duplicates of ilm-move-to-step **URL:** \ **Category:** Elasticsearch\ **Created:** [January 31, 2023, 12:55pm UTC](https://discuss.elastic.co/t/pending-tasks-has-millions-of-entries-many-of-with-are-exact-duplicates-of-ilm-move-to-step/324329 "2023-01-31T12:55:39Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![jmlucjav](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jmlucjav/32/5870_2.png) [@jmlucjav](https://discuss.elastic.co/u/jmlucjav)\ **Post date:** [January 31, 2023, 12:55pm UTC](https://discuss.elastic.co/t/pending-tasks-has-millions-of-entries-many-of-with-are-exact-duplicates-of-ilm-move-to-step/324329/1 "2023-01-31T12:55:39Z")
hi, I have a 7.10.2 cluster, quite large, with 10k indices, and we are having issues with millions of pending tasks being queued at some point. I managed to get a dump of them while they were just 2M, and I saw: - 90% of them are ilm-move-to-step - for a given ilm-move-to-step, we can see it repeats \> 2k times like so: ```auto 20439 1.5h NORMAL ilm-move-to-step {policy [retention_4d], index [index1234], currentStep [{"phase":"new","action":"complete","name":"complete"}], nextStep [{"phase":"warm","action":"set_priority","name":"set_priority"}]} 38435 1.5h NORMAL ilm-move-to-step {policy [retention_4d], index [index1234], currentStep [{"p... ``` This cluster has 45 data nodes. What could be the reason for so many duplication in tasks? many thanks ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [January 31, 2023, 3:31pm UTC](https://discuss.elastic.co/t/pending-tasks-has-millions-of-entries-many-of-with-are-exact-duplicates-of-ilm-move-to-step/324329/2 "2023-01-31T15:31:18Z")
> [@jmlucjav](#): > > I have a 7.10.2 cluster This version is very old, long past EOL. You need to upgrade to a supported version as soon as possible. There are several known scalability issues in this version that could explain these symptoms, and which are fixed in newer versions. See e.g. the issues linked from here: > > > This meta issue tracks known issues with scaling clusters to large numbers of sh…ards. > > \- Security > - \[\] https://github.com/elastic/elasticsearch/issues/67987 > - \[x\] https://github.com/elastic/elasticsearch/issues/79632 > > \- General > - \[\] https://github.com/elastic/elasticsearch/issues/51992 > - \[\] https://github.com/elastic/elasticsearch/issues/87555 > - \[x\] https://github.com/elastic/elasticsearch/pull/77546 > - \[\] https://github.com/elastic/elasticsearch/issues/79563 > - \[x\] https://github.com/elastic/elasticsearch/pull/79793 > - \[x\] https://github.com/elastic/elasticsearch/issues/79906 > - \[x\] https://github.com/elastic/elasticsearch/pull/80064 > - \[\] https://github.com/elastic/elasticsearch/issues/80694 > - \[x\] https://github.com/elastic/elasticsearch/issues/81626 > - \[x\] https://github.com/elastic/elasticsearch/issues/81627 > - \[x\] https://github.com/elastic/elasticsearch/issues/81628 > - \[\] https://github.com/elastic/elasticsearch/issues/81846 > - \[x\] https://github.com/elastic/elasticsearch/issues/81880 > - \[x\] https://github.com/elastic/elasticsearch/issues/82337 > - \[x\] https://github.com/elastic/elasticsearch/issues/82342 > - \[x\] https://github.com/elastic/elasticsearch/pull/82608 > - \[x\] https://github.com/elastic/elasticsearch/pull/82227 > - \[x\] https://github.com/elastic/elasticsearch/pull/82830 > - \[\] https://github.com/elastic/elasticsearch/issues/83049 > - \[\] https://github.com/elastic/elasticsearch/issues/83203 > - \[x\] https://github.com/elastic/elasticsearch/issues/83204 -\> #85380 > - \[x\] https://github.com/elastic/elasticsearch/issues/85839 > - \[x\] https://github.com/elastic/elasticsearch/issues/86639 > - \[x\] https://github.com/elastic/elasticsearch/issues/87681 > - \[\] https://github.com/elastic/elasticsearch/issues/90631 > - \[x\] https://github.com/elastic/elasticsearch/pull/92902 > - \[\] https://github.com/elastic/elasticsearch/issues/97222 > - \[x\] https://github.com/elastic/elasticsearch/issues/97914 > - \[\] https://github.com/elastic/elasticsearch/issues/101805 > > \- Snapshots + SLM > - \[x\] https://github.com/elastic/elasticsearch/pull/80942 > - \[\] https://github.com/elastic/elasticsearch/issues/82824 > - \[x\] https://github.com/elastic/elasticsearch/issues/82853 > - \[x\] https://github.com/elastic/elasticsearch/issues/82937 > - \[x\] https://github.com/elastic/elasticsearch/pull/88707 > - \[x\] https://github.com/elastic/elasticsearch/issues/88732 > - \[\] https://github.com/elastic/elasticsearch/issues/89163 > - \[x\] https://github.com/elastic/elasticsearch/issues/89952 > - \[x\] https://github.com/elastic/elasticsearch/issues/99953 > > \- Metrics > - \[x\] https://github.com/elastic/elasticsearch/issues/80428 > > \- ILM + Allocation > - \[x\] https://github.com/elastic/elasticsearch/pull/78547 > - \[x\] #78235 > - \[x\] https://github.com/elastic/elasticsearch/issues/78246 > - \[x\] #78075 > - \[x\] #77965 > - \[x\] https://github.com/elastic/elasticsearch/pull/77855 > - \[x\] https://github.com/elastic/elasticsearch/pull/77863 > - \[x\] https://github.com/elastic/elasticsearch/pull/78742 > - \[x\] https://github.com/elastic/elasticsearch/pull/78668 > - \[x\] https://github.com/elastic/elasticsearch/pull/78672 > - \[x\] https://github.com/elastic/elasticsearch/pull/78609 > - \[x\] https://github.com/elastic/elasticsearch/pull/78745 > - \[x\] https://github.com/elastic/elasticsearch/pull/78813 > - \[\] https://github.com/elastic/elasticsearch/issues/78892 > - \[x\] https://github.com/elastic/elasticsearch/pull/80493 > - \[x\] https://github.com/elastic/elasticsearch/issues/77888 > - \[x\] https://github.com/elastic/elasticsearch/pull/78931 > - \[x\] https://github.com/elastic/elasticsearch/pull/78969 > - \[x\] https://github.com/elastic/elasticsearch/issues/78980 > - \[x\] https://github.com/elastic/elasticsearch/issues/79782 > - \[x\] https://github.com/elastic/elasticsearch/issues/79866 > - \[x\] https://github.com/elastic/elasticsearch/pull/79860 > - \[x\] https://github.com/elastic/elasticsearch/pull/79941 > - \[x\] https://github.com/elastic/elasticsearch/pull/80179 > - \[\] https://github.com/elastic/elasticsearch/issues/80407 > - \[x\] https://github.com/elastic/elasticsearch/issues/81880 > - \[x\] https://github.com/elastic/elasticsearch/pull/82251 > - \[x\] https://github.com/elastic/elasticsearch/issues/82708 > - \[x\] https://github.com/elastic/elasticsearch/pull/83092 > - \[x\] https://github.com/elastic/elasticsearch/pull/83241 > - \[x\] https://github.com/elastic/elasticsearch/pull/83340 > - \[x\] https://github.com/elastic/elasticsearch/issues/83582 > - \[x\] https://github.com/elastic/elasticsearch/pull/84034 > - \[x\] https://github.com/elastic/elasticsearch/issues/89924 > - \[\] https://github.com/elastic/elasticsearch/issues/94933 > - \[x\] https://github.com/elastic/elasticsearch/issues/100230 > > \- Search > - \[x\] https://github.com/elastic/elasticsearch/issues/74648 > - \[x\] https://github.com/elastic/elasticsearch/issues/78164 > - \[x\] https://github.com/elastic/elasticsearch/pull/76405 > - \[x\] https://github.com/elastic/elasticsearch/pull/77131 > - \[x\] https://github.com/elastic/elasticsearch/pull/77201 > - \[x\] https://github.com/elastic/elasticsearch/pull/77251 > - \[x\] https://github.com/elastic/elasticsearch/issues/78314 > - \[x\] https://github.com/elastic/elasticsearch/pull/78508 > - \[x\] https://github.com/elastic/elasticsearch/issues/82879 > - \[\] https://github.com/elastic/elasticsearch/issues/89309 > - \[x\] https://github.com/elastic/elasticsearch/issues/90622 > - \[\] https://github.com/elastic/elasticsearch/pull/99219 > - \[\] https://github.com/elastic/elasticsearch/issues/99220 > - \[\] https://github.com/elastic/elasticsearch/pull/99222 > > \- Network > - \[x\] https://github.com/elastic/elasticsearch/issues/82245 > - \[x\] https://github.com/elastic/elasticsearch/issues/79560 > - \[\] https://github.com/elastic/elasticsearch/pull/83846 > - \[\] https://github.com/elastic/elasticsearch/issues/84876 > - \[\] https://github.com/elastic/elasticsearch/issues/84887 > - \[\] https://github.com/elastic/elasticsearch/issues/89838 ---
**Author:** ![jmlucjav](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jmlucjav/32/5870_2.png) [@jmlucjav](https://discuss.elastic.co/u/jmlucjav)\ **Post date:** [January 31, 2023, 3:54pm UTC](https://discuss.elastic.co/t/pending-tasks-has-millions-of-entries-many-of-with-are-exact-duplicates-of-ilm-move-to-step/324329/3 "2023-01-31T15:54:57Z")
hi David, sure it's old, but upgrading it's not so easy. Do you reckon upgrading to 7.17 would bring up some benefits? Upgrading to 8 it's a much harder thing, but 7.17 we might do... thanks! ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [January 31, 2023, 4:27pm UTC](https://discuss.elastic.co/t/pending-tasks-has-millions-of-entries-many-of-with-are-exact-duplicates-of-ilm-move-to-step/324329/4 "2023-01-31T16:27:09Z")
Yes, 7.17 will give you some of the benefits I linked. ---
**Author:** ![jmlucjav](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jmlucjav/32/5870_2.png) [@jmlucjav](https://discuss.elastic.co/u/jmlucjav)\ **Post date:** [January 31, 2023, 6:56pm UTC](https://discuss.elastic.co/t/pending-tasks-has-millions-of-entries-many-of-with-are-exact-duplicates-of-ilm-move-to-step/324329/5 "2023-01-31T18:56:58Z")
much appreciated! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 28, 2023, 6:57pm UTC](https://discuss.elastic.co/t/pending-tasks-has-millions-of-entries-many-of-with-are-exact-duplicates-of-ilm-move-to-step/324329/6 "2023-02-28T18:57:34Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.