Indices not getting an ILM policy applied after rollover

[jba (Jan Bruun Andersen)]

I have a problem with indices not getting an ILM policy applied after rollover on a cluster (Elasticsearch, Logstash, and Kibana) that have recently been upgraded from 7.17 to 8.4 and have had our old legacy templates concerted to the new Composable-compatible format (converted mechanical using a small jq script).

The rollover works fine in our production cluster (without the Composable-templates installed, still using legacy templates) but in all our test-cluster (where we have deployed the new-format templates) we have the problem of no ILM policy being applied to rolled-over indices.

My question is this (based on my incomplete understanding of the whole stack):

What is responsible for creating the new index and applying the ILM policy?

Is it the ILM component that creates the new fresh index and is responsible for copying over the existing policy?

Or is it Logstash and the logstash-elasticsearch-output plugin that, when configured with manage_template => true and/or ilm_enabled => true, figures out that it is supposed to apply the ILM policy to a fresh index when the template itself does not specify a policy?

Or is it something else?

[stephenb (Stephen Brown)]

@jba

manage_template => true and/or ilm_enabled => true, f

If you are fully managing your templates take those out.

@jba

Is it the ILM component that creates the new fresh index and is responsible for copying over the existing policy?

Policies do not get copied they get applied to the newly created index.
The ILM policy triggers a rollover
The Rollover Action creates a new index
The Correct Template gets applied if it fits the index pattern and it is the highest priority
The Template Specifies the ILM Policy
The Template Specifies the Rollover Alias

[jba (Jan Bruun Andersen)]

Thanks for your answer, Stephen.

I am not sure that I can just remove the manage_template and ilm_enabled settings from our Logstash configurations. We have had those settings there since forever, and it worked fine until we upgraded to 8.4.1 and started replacing the legacy templates with the new composable template format. If I remove them we need to:

1. Change the way we automatically deploy templates

2. Figure out how to apply ILM policies with different max_age for the rollover and deletion phases (we have different retention policies depending if we are in a production cluster, a test cluster, or development cluster)

I and sure that it is doable, but as I wrote, it works in production where we not yet dared switching over to the new composable template format.

I think that the second part of your reply have the clue to solving our problem with missing ILM policies on newly rolled-over indices. I think we need to increase the priority of our own templates to ensure that they get picked instead of a generic template with no attached ILM policy.

[stephenb (Stephen Brown)]

@jba

I am not sure that I can just remove the manage_template and ilm_enabled settings from our Logstash configurations. We have had those settings there since forever, and it worked fine until we upgraded to 8.4.1 and started replacing the legacy templates with the new composable template format. If I remove them we need to:

Understood you are probably still using some of the default logstash indices / templates etc ... my caveat is / was if you are fully managing your templates, indices etc ... not using any defaults is what I meant. i.e. every index that is output uses a dedicated template etc..etc..

And yes ... certainly do not break production