Assigning an Index Template to an ILM Policy Gets Periodically Removed

Elastic Stack Kibana
1

As the title suggests, I keep assigning the index template that I want to be linked with a certain ILM, then after some time I go check it and find that they went back to the previous one.

image
image1670×511 36.7 KB

This happened to me numerous times, and I can't find any related problem like it online. Idk why. Would really appreciate the support.

2

I encountered a similar issue when the corresponding template is generated by an Integration (over Fleet).

Alle components of an integration (so templates, ingest pipelines, etc.) are periodically refreshed from the definition in the integration.

To change these there is in most cases a dedicated name of the corresponding component to customize these (in ingest pipelines these are named with "@custom" at the end).

I don't know if there is a similar mechenic for index template or if that is something to be done in the integration configuration.

3

Please, provide more context, what template are you changing?

If you make changes to any managed template they may not persist and be reverted on integration updates.

If you want to make any changes to data that is ingested using any of Elastic Integrations, you need to make the changes on the equivalent @custom component template.

The @custom component templates will persist during integration updates.

There are 3 levels of custom settings on those templates, one that affects all integrations on the logs@custom component template, one that will affect all datasets in the integration, normallly the integration_name@custom and one that will affect only a specific dataset in the integration, which will be on something like logs-datastrream.dataset@custom.

Those @custom template does not exist per default, you need to create them by selecting the main index template and them clicking on the component template name, like this:

image
image752×679 47.6 KB

This will open the interface for you to edit the template.

4

Thanks @Shaoranlaos I will check it

@leandrojmp Thanks for the reply

Normally I move any integration index, let's say, logs-system.syslog-default to a customly made logs@customILM

I'm not changing a template itself, I'm only adding it to the ILM policy.

So like I have said, periodically, it will move back to logs@lifecycle or the deprecated logs ILM.

If I understand correctly, a component template does this, and I will have to create a custom component template and add it to the ILM I want, is that what you're saying?

Appreciate the help!

5

This is a change, adding an ILM policy to a template will change the setting in that template, which can be reverted on an integration update.

You should not make any changes to any managed index template, all customizations needs to be done using the @custom component template.

If you want to change the ILM policy for the data stream logs-system.syslog-default, then you need to edit the custom component template logs-system.syslog@custom and add something like this in the settings:

{
  "index": {
    "lifecycle": {
      "name": "policy-name"
    }
  }
}
6

Got it, now I understand, thank you.

Another related question is, what if I want to automate this? I tried doing this with the _index_template API endpoint and if did something like that it will overwrite the entire template which is something we do not want.

7

You just need to use the correct API, for component templates it is _component_template.

Then you could use something like this:

PUT _component_template/logs-system.syslog@custom
{ component template payload }

You can do this process through the UI and in the review part you can see what will be the final request.

To change just the ILM policy, it would be something like this:

PUT _component_template/logs-system.syslog@custom
{
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "policy-name"
        }
      }
    }
  }
}
8

Thank you, but my question is automating putting the components templates onto the templates that I want to add the ILM onto.

So I want to also update logs-system.auth as well for example and many others. So going to each one isn't really optimal, so I tried adding them using the API endpoint index_template overwrites the previous template.

9

It depends, if you want to use the same ILM policy in all datasets for an integration, you can make the customization on the package level as mentioned, if you want logs-system.auth to use one ILM policy and logs-system.syslog to use a different policy, then you will need to edit the component templates per dataset, this is how this was build.

For example, considering the system integration.

image
image453×481 26.6 KB

If you want to change any settings for just the syslog dataset, you use the logs-system.syslog@custom, if you want to change for all datasets in the system integration, you use system@custom and if you want to change for all datasets in all logs integrations, you use logs@custom.

The following request will change the policy for logs-system.syslog, logs-system.security, logs-system.system, logs-system.auth and logs-system.applications

PUT _component_template/system@custom
{
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "policy-name"
        }
      }
    }
  }
}