# ILM slows down node recovery/rolling upgrade process **URL:** \ **Category:** Elasticsearch\ **Tags:** ilm-index-lifecycle-management\ **Created:** [March 29, 2024, 8:12pm UTC](https://discuss.elastic.co/t/ilm-slows-down-node-recovery-rolling-upgrade-process/356476 "2024-03-29T20:12:07Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![jerrac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerrac/32/52980_2.png) [@jerrac](https://discuss.elastic.co/u/jerrac)\ **Post date:** [March 29, 2024, 8:12pm UTC](https://discuss.elastic.co/t/ilm-slows-down-node-recovery-rolling-upgrade-process/356476/1 "2024-03-29T20:12:07Z")
On my test cluster, I started a rolling upgrade. It's a 3 node cluster, so the cluster state went promptly to red. Once my upgraded node started up, I found that the index reallocation/recovery process was taking a lot longer than it should. As in minutes per shard. I also noticed a lot of ILM related errors in the logs. Putting two and two together, I stopped ILM and the recovery process started speeding up after a bit. I'm guessing it had to wait for the ILM jobs that were active to fully stop. Which leaves me wondering, why did ILM even try to do anything when the cluster was red? Even if that wasn't a sign to stop, I'd think finding that the shards needed for the job are unallocated or not recovered would trigger a quick short circuit. (I'm assuming the logs I'm seeing are related to the unallocated shards, I haven't actually confirmed that.) Why would it keep trying to do enough work that allocation would slow down so drastically? For older version of ES there were instructions about disabling shard reallocation to speed up recovery during a rolling upgrade, but those instructions seem to be gone. As far as [I can tell](https://www.elastic.co/guide/en/elastic-stack/8.13/upgrading-elastic-stack.html#prepare-to-upgrade-8x) there are no special rolling upgrade instructions anymore. So I don't think I'm missing anything, but am I? Or is this just something that happens, and the solution is to remember to stop ILM before upgrading? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [March 30, 2024, 8:41am UTC](https://discuss.elastic.co/t/ilm-slows-down-node-recovery-rolling-upgrade-process/356476/2 "2024-03-30T08:41:18Z")
> [@jerrac](#): > > so the cluster state went promptly to red It sounds like you expected this, but that's not right. The cluster health should be `yellow` or `green`, never `red` during a rolling restart/upgrade. You must have had some unassigned shards, or shards without replicas, to cause that. > [@jerrac](#): > > why did ILM even try to do anything when the cluster was red? Red health just means that at least one primary shard is unassigned. That's not a reason for ILM to stop, and indeed we want ILM to keep on trying even on the unhealthy index so that it can proceed ASAP when the index returns to health. Manually stopping ILM shouldn't be necessary during a rolling upgrade. Could you share the output of `GET _cluster/stats?pretty` please? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [March 30, 2024, 9:56am UTC](https://discuss.elastic.co/t/ilm-slows-down-node-recovery-rolling-upgrade-process/356476/3 "2024-03-30T09:56:31Z")
> [@jerrac](#): > > For older version of ES there were instructions about disabling shard reallocation to speed up recovery during a rolling upgrade, but those instructions seem to be gone. As far as [I can tell](https://www.elastic.co/guide/en/elastic-stack/8.13/upgrading-elastic-stack.html#prepare-to-upgrade-8x) there are no special rolling upgrade instructions anymore. There's still a recommended process to follow for a [rolling upgrade](https://www.elastic.co/guide/en/elastic-stack/8.13/upgrading-elasticsearch.html) (effectively the same as a [rolling restart](https://www.elastic.co/guide/en/elasticsearch/reference/current/restart-cluster.html#restart-cluster-rolling)). But disabling ILM is not part of this process. ---
**Author:** ![jerrac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerrac/32/52980_2.png) [@jerrac](https://discuss.elastic.co/u/jerrac)\ **Post date:** [March 31, 2024, 4:57pm UTC](https://discuss.elastic.co/t/ilm-slows-down-node-recovery-rolling-upgrade-process/356476/4 "2024-03-31T16:57:49Z")
> [@DavidTurner](#): > > It sounds like you expected this, but that's not right. The cluster health should be `yellow` or `green`, never `red` during a rolling restart/upgrade. You must have had some unassigned shards, or shards without replicas, to cause that. Thanks to lack of resources, I have ES configured to only have a single primary shard for each index. So that would explain the red status. (I originally tried to only have 1 node as well, but restarts on that got slow really fast due to needing to open more indexes than normal.) Though, I did think taking 1 node out of a 3 node cluster would make it go red as well. > [@DavidTurner](#): > > Manually stopping ILM shouldn't be necessary during a rolling upgrade. Could you share the output of `GET _cluster/stats?pretty` please? By the time I'm back in the office, it'll have been a while. Is it possible to load the stats for a specific time period? > [@DavidTurner](#): > > There's still a recommended process to follow for a [rolling upgrade](https://www.elastic.co/guide/en/elastic-stack/8.13/upgrading-elasticsearch.html) (effectively the same as a [rolling restart](https://www.elastic.co/guide/en/elasticsearch/reference/current/restart-cluster.html#restart-cluster-rolling)). But disabling ILM is not part of this process. Ok, sheesh. I looked for that doc repeatedly over the past few versions and somehow missed it. So I should still stop reallocation between nodes. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [March 31, 2024, 6:44pm UTC](https://discuss.elastic.co/t/ilm-slows-down-node-recovery-rolling-upgrade-process/356476/5 "2024-03-31T18:44:53Z")
> [@jerrac](#): > > I have ES configured to only have a single primary shard for each index. So that would explain the red status By "single primary shard" do you mean `number_of_replicas: 0`? In which case are you sure a rolling upgrade makes sense? It's quite a complex sequence of steps designed to maintain availability throughout, but that requires replicas - if you are ok with a period of unavailability then it'd be a good deal simpler (and possibly quicker) to upgrade all the nodes at the same time. > [@jerrac](#): > > By the time I'm back in the office, it'll have been a while. Is it possible to load the stats for a specific time period? No, but the current stats will still be useful to see. ---
**Author:** ![jerrac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerrac/32/52980_2.png) [@jerrac](https://discuss.elastic.co/u/jerrac)\ **Post date:** [April 1, 2024, 9:53pm UTC](https://discuss.elastic.co/t/ilm-slows-down-node-recovery-rolling-upgrade-process/356476/6 "2024-04-01T21:53:27Z")
> [@DavidTurner](#): > > No, but the current stats will still be useful to see. Any place I can send the stats that isn't public? I'm feeling overly cautious today. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [April 2, 2024, 5:48am UTC](https://discuss.elastic.co/t/ilm-slows-down-node-recovery-rolling-upgrade-process/356476/7 "2024-04-02T05:48:46Z")
You can send me a DM if you'd prefer. ---
**Author:** ![jerrac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerrac/32/52980_2.png) [@jerrac](https://discuss.elastic.co/u/jerrac)\ **Post date:** [April 2, 2024, 4:02pm UTC](https://discuss.elastic.co/t/ilm-slows-down-node-recovery-rolling-upgrade-process/356476/8 "2024-04-02T16:02:51Z")
Er, you have DM's turned off. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [April 2, 2024, 5:37pm UTC](https://discuss.elastic.co/t/ilm-slows-down-node-recovery-rolling-upgrade-process/356476/9 "2024-04-02T17:37:29Z")
Not sure what's wrong with my DMs but we connected eventually. I don't see anything particularly concerning unfortunately. A decent shard count but not ludicrously high, and a pretty recent version: ```auto "indices": { "count": 7626, "shards": { "total": 7700, "primaries": 7626, ... "versions": [ "8.12.0" ], ``` I think if ILM really is slowing things down so much when the cluster has many unassigned primaries then we would consider that a bug. It's not a scenario that we cover in any benchmarks today AFAIK so it wouldn't surprise me if there were a performance bug of this nature. Pretty tricky to know how to fix it tho. Maybe collecting `GET _nodes/_master/hot_threads?threads=9999` repeatedly while the cluster is recovering would help shed some light on the problem. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [April 9, 2024, 5:51pm UTC](https://discuss.elastic.co/t/ilm-slows-down-node-recovery-rolling-upgrade-process/356476/10 "2024-04-09T17:51:37Z")
I looked at some hot threads captured during recovery (also shared via DM) but I didn't see anything obviously related to ILM in there. It caught one snapshot that's perhaps a little suspicious: ```auto java.base@21.0.2/java.lang.Throwable.(Throwable.java:273) java.base@21.0.2/java.lang.Exception.(Exception.java:67) java.base@21.0.2/java.lang.RuntimeException.(RuntimeException.java:63) app/org.elasticsearch.server@8.13.2/org.elasticsearch.ElasticsearchException.(ElasticsearchException.java:134) app/org.elasticsearch.server@8.13.2/org.elasticsearch.ResourceAlreadyExistsException.(ResourceAlreadyExistsException.java:26) app/org.elasticsearch.server@8.13.2/org.elasticsearch.ResourceAlreadyExistsException.(ResourceAlreadyExistsException.java:21) app/org.elasticsearch.server@8.13.2/org.elasticsearch.cluster.metadata.MetadataCreateIndexService.validateIndexName(MetadataCreateIndexService.java:177) app/org.elasticsearch.server@8.13.2/org.elasticsearch.cluster.metadata.MetadataCreateIndexService.validate(MetadataCreateIndexService.java:1370) app/org.elasticsearch.server@8.13.2/org.elasticsearch.cluster.metadata.MetadataCreateIndexService.applyCreateIndexRequest(MetadataCreateIndexService.java:345) org.elasticsearch.xpack.downsample.TransportDownsampleAction$3.execute(TransportDownsampleAction.java:800) org.elasticsearch.xpack.downsample.TransportDownsampleAction$1.executeTask(TransportDownsampleAction.java:136) org.elasticsearch.xpack.downsample.TransportDownsampleAction$1.executeTask(TransportDownsampleAction.java:132) app/org.elasticsearch.server@8.13.2/org.elasticsearch.cluster.SimpleBatchedExecutor.execute(SimpleBatchedExecutor.java:70) ``` Maybe that's ILM triggering an unnecessary downsample action? Not sure. I think I'd like to see debug logs from the `MasterService` (i.e. set `logger.org.elasticsearch.cluster.service.MasterService: DEBUG`) to get a sense of what cluster state updates it's processing. Maybe there's substantially more of them when ILM is running than otherwise. ---
**Author:** ![jerrac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerrac/32/52980_2.png) [@jerrac](https://discuss.elastic.co/u/jerrac)\ **Post date:** [April 9, 2024, 6:32pm UTC](https://discuss.elastic.co/t/ilm-slows-down-node-recovery-rolling-upgrade-process/356476/11 "2024-04-09T18:32:09Z")
> [@DavidTurner](#): > > I think I'd like to see debug logs from the `MasterService` (i.e. set `logger.org.elasticsearch.cluster.service.MasterService: DEBUG`) to get a sense of what cluster state updates it's processing. Maybe there's substantially more of them when ILM is running than otherwise. So, that'd be the logs during the recovery, right? Is that an elasticsearch.yml setting, or somewhere else? ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [April 9, 2024, 7:37pm UTC](https://discuss.elastic.co/t/ilm-slows-down-node-recovery-rolling-upgrade-process/356476/12 "2024-04-09T19:37:44Z")
You can put it in `elasticsearch.yml` but normally you'd set it as a persistent cluster setting. See [these docs](https://www.elastic.co/guide/en/elasticsearch/reference/current/logging.html#configuring-logging-levels) for more info. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [May 7, 2024, 7:06pm UTC](https://discuss.elastic.co/t/ilm-slows-down-node-recovery-rolling-upgrade-process/356476/13 "2024-05-07T19:06:50Z")
@jerrac thanks for sharing the debug logs from another cluster restart this morning. I can't see the symptoms you were originally asking about - the master appears to be starting up one or two batches of shards every second, and it only takes 15 min or so to bring all ~7k indices back up from cold. ---
**Author:** ![jerrac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerrac/32/52980_2.png) [@jerrac](https://discuss.elastic.co/u/jerrac)\ **Post date:** [May 7, 2024, 8:04pm UTC](https://discuss.elastic.co/t/ilm-slows-down-node-recovery-rolling-upgrade-process/356476/14 "2024-05-07T20:04:36Z")
Thanks for checking. 🙂 I did see a few spots where the cat/health output was saying it was taking minutes to do an action (reallocation I think? Can't remember the meaning of those columns of the top of my head...) But it did eventually get back up to green. So I'm just going to let this one go. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 4, 2024, 8:04pm UTC](https://discuss.elastic.co/t/ilm-slows-down-node-recovery-rolling-upgrade-process/356476/15 "2024-06-04T20:04:38Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.