Change template and ILM policy from an existing Datastream

[Angelos]

Hey!

In our Elasticsearch we use Datastreams which split our data to several streams like bellow.

So all the Datastreams have the same template and ILM policy. Since one of them has increased a lot I want to apply different ILM policy so I can delete data.

How can I change the ILM policy from only one of those templates, for example prod-data-metric2-ds?

Our index_template:

Index pattern
   prod-data-*

Priority
    200

Component templates
    None

Data stream
    Yes

Version
    None

{
  "template": {
    "settings": {
      "index": {
        "lifecycle": {
          "name": "prod_data_metrics_policy"
        },
        "search": {
          "slowlog": {
            "level": "info",
            "threshold": {
              "fetch": {
                "warn": "-1",
                "trace": "-1",
                "debug": "-1",
                "info": "-1"
              },
              "query": {
                "warn": "60s",
                "trace": "0ms",
                "debug": "0ms",
                "info": "0ms"
              }
            }
          }
        },
        "refresh_interval": "60s",
        "number_of_shards": "3",
        "number_of_replicas": "1"
      }
    },
    "aliases": {},
    "mappings": {}
  }
}

Datastreams:

prod-data-metric1-ds
prod-data-metric2-ds
prod-data-metric3-ds
prod-data-metric4-ds

/Angelos

[leandrojmp (Leandro Pereira)]

If you are using component templates, you will need a new template with a higher priority number.

This new template should match only the desired data-stream, for example prod-data-metric2-*.

If you are using legacy templates you will also need a new template matching only the desired data-stream, but in this case you will need it to have a higher order value.

[Angelos]

I am using Index Template and it is configured to have

index pattern: 
prod-data-*

ILM policy:
prod_data_metrics_policy

If I create a new index template with

index pattern:
prod-data-metric2-*

ILM policy:
prod_data_metric2_policy

won't I have conflicts with duplicate datastreams with different templates?
Since prod-data-metric2-ds is included in both prod-data-metric2-* and prod-data-*

[leandrojmp (Leandro Pereira)]

You will need to use a higher priority or order for the specific template, depending if you are using the component templates or the legacy template.

A template with a higher order or priority will override the settings of a template with a lower order or priority.

[Angelos]

Thank you for your fast responses!
I have Lifecycle errors:

Index lifecycle error
exception: Concurrent modification of alias [prod-data-metric2-ds] during rollover

Should I put rollover_alias in the template?
If I do that should I create an initial index?

PUT test-000001
{
  "aliases": {
    "test-alias":{
      "is_write_index": true 
    }
  }
}

The problem is that there are already indices for that datastream.

[Angelos]

@leandrojmp It seems the latest implementation works but I still have a lot lifecycling errors and many new empty indices.

Index lifecycle error
exception: Concurrent modification of alias [prod-data-metric2-ds] during rollover

The errors are being reduced but very slowly, is this normal?

/Angelos

[leandrojmp (Leandro Pereira)]

If your policy has a rollover you can't apply it manually to index, you need to use templates.

Check this answer to your other post.

When you have a rollover in a policy, do not change the policies manually, change it in the templates and let the template take care of the change.

[Angelos]

So now that I have done that, is there any way I can fix it or should I wait until all the errors will be resolved.

/Angelos

[Angelos]

To solve the errors I removed the ILM policy from all the indices, recreate the policy and applied the new policy to the latest index.
Now the rollover works correctly with the data ingestion.

Note that when I tried to remove the policy targeting the Datastream,

POST prod-data-metric2-ds/_ilm/remove

didn't remove it from all the indices. I needed to remove it manually from each remaining index.