Unable to update ILM policy

Elastic Stack Elasticsearch
1

Hello,

I have a hot-warm architecture cluster running in production containing 1000+ daily indices spanning 3 months. A single ILM policy moves indices from hot to warm after 4 days of creation, then to cold phase after 30 days and delete phase after 90 days.

Currently all of these indices have this ILM policy applied directly via templates.

Warm phase is configured to move an index from hot nodes to warm nodes. Problem here is that warm nodes are running out of disk space where as hot nodes have ample space left.

Hence I attempted to update the warm phase timing from 4 days to say, 6 days to increase retention on hot nodes. But the cluster doesn't seem to accept this change either via Kibana or via REST API. This could be because there are indices currently executing this phase.

What are my options here apart from deleting old indices or adding more disk?

Any help would be appreciated, thanks!

1 Like
2

Hello team,
Any response would be appreciated!

3

Hi @anirudhra10. I had this same question. I'll be posting a response shortly.

4

You are correct. Per Lifecycle policy updates | Elasticsearch Guide [8.11] | Elastic, an index keeps a copy of the current phase's policy once it has started executing (or waiting for the condition in your case.)

There is the ILM explain API end point that I was using to list the version of the policy currently being used for a given index, which can assisted in troubleshooting.

Unfortunately, I've got to log off now. I'll come back with some recommendations soon.

5

Sorry, about the interruption in my response.

There are a couple of different options. I have not done this in production but this worked for me the other day in test.

  1. Remove the ILM policy from the indices
    2.Update the settings on the index to re-filter the index shards back to the hot nodes
  2. Add the updated policy to the index
  3. Use the explain API to check that the index is back in the hot (aka init) stage and will transition as expected
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.