Index lifecycle management for existing indices?

[asp]

Hi all,

I am currently checking out the index lifecycle management in 6.6.2.

As I understand the docs I can add a policy to an index at index creation or indirectly by an index template.

1. How can I add / update to an index which is already existing?
2. What happens if I change the policy after the index is created? Will the change be valid for existing indices because the policy is linked to the index, or does the index contain just a one time copy of the policy?
3. How can I delete the reference between index or index template and ILM policy?

Thanks a lot, Andreas

[asp]

does anyone have an idea?

[gbrown (Gordon Brown)]

@asp

• How can I add / update to an index which is already existing?

You can have ILM manage an existing index by updating the index settings with "index.lifecycle.name": "my-policy", where my-policy is the policy you want to use. The index will then be managed by the policy from the start of the policy.

Note that if you're applying a policy to historical indices, you will likely want to use a policy that does not include the rollover action for those indices.

@asp

• What happens if I change the policy after the index is created? Will the change be valid for existing indices because the policy is linked to the index, or does the index contain just a one time copy of the policy?

Updating a policy which is in use for at least one index works, but the index will continue to use the old policy until it enters a new phase (e.g. goes from hot to warm or warm to cold). See the Updates to executing policies page in the docs for details.

@asp

• How can I delete the reference between index or index template and ILM policy?

To remove a policy from an index, use the Remove Policy from Index API. To remove a policy from an index template, just remove the index.lifecycle.name setting from the template.

[asp]

Hi @gbrown, thanks for the reply first.

@gbrown

• What happens if I change the policy after the index is created? Will the change be valid for existing indices because the policy is linked to the index, or does the index contain just a one time copy of the policy?

Updating a policy which is in use for at least one index works, but the index will continue to use the old policy until it enters a new phase (e.g. goes from hot to warm or warm to cold ). See the Updates to executing policies page in the docs for details.

That is something what I feared of:
Lets say I have a policy created which rolls over after 20GB, moves from hot to cold after 30 days and deletes after 30 days in status cold. I apply that for my indices.

Then I see, damn, I made a mistake (copy and paste error). It should be rollover after 20GB, move to cold after 7 days and delete cold after 30 days.
As I understand for existing indices which are already in hot phase, they will be kept for 30 days and only new indices will stay 7 days in hot phase.

And I can just see, that all my indices are using my-policy which is now configured to 7 days cold.
Is my understanding correct?

So the only solution would be to update all indexes settings to delete the policy settings and re-add the policy to take effect immediately, or would that also not work?

Is it possible for next version to give the option to say:

• update only for new indices
• update for all indices (also for existing indices)

?

[gbrown (Gordon Brown)]

@asp

As I understand for existing indices which are already in hot phase, they will be kept for 30 days and only new indices will stay 7 days in hot phase.

Any indices which are still waiting to enter the cold phase will use the updated policy, but any that have already entered the cold phase will use the cold phase defined in the old version of the policy. In your case, if you have an index that rolled over 8 days ago, then you update the min_age for the cold phase in your policy to 7 days, the index will then immediately move to the cold phase.

The reason why this works the way it does is that, if an index is in the middle of a phase, and the phase was updated to use a completely different set of actions, ILM wouldn't know what to do, so if the policy is updated while an index is partway through a phase it will just keep using the same one.

@asp

And I can just see, that all my indices are using my-policy which is now configured to 7 days cold.
Is my understanding correct?

If I'm understanding you correctly, you can see which indices are using the old version by using the Explain API, as shown in the article I linked to - the phase_execution section of the response contains both the version of the policy in use and the actual definition of the phase from the old version of the policy that the index is still using. If the index enters a different phase from the one returned in that response, it will get the new phase from the most recent version of the policy at the time it moves to the new phase.

@asp

So the only solution would be to update all indexes settings to delete the policy settings and re-add the policy to take effect immediately, or would that also not work?

If you need to change the policy for an index and don't want to wait until the index is through its current phase, then yes, you can remove the policy (although you need to use the Remove policy API rather than just setting index.lifecycle.name to null as the remove policy API will clear the metadata Elasticsearch uses to keep track of where an index is in its policy. (aside: there's an open issue on Github to remove the need for the remove policy API)

Note that when you re-add the policy, the indices will all start from the beginning of the policy, so if you're re-adding a policy to indices that have already rolled over, you probably want to use a policy that does not have rollover.

@asp

Is it possible for next version to give the option to say:

• update only for new indices

What I'd recommend for this use case is to create a new policy (for example, my-policy-version-2) and change your index template to use the new policy, rather than updating the existing policy. If you have use cases with slightly different needs, even for the same "set" of indices, don't be afraid to create multiple policies to handle the different or changing requirements.

@asp

update for all indices (also for existing indices)

We are very aware that the current behavior isn't intuitive when it comes to updating existing indices - we would very much like to improve it, but the problem of what to do when an index is partway through a phase when the policy is updated is a difficult one. We're working on it, though!