Index template alias not being applied to backing indexes of datastream

Elastic version 8.17.1

Context: multiple systems logging to a datastream, so we want to create one dataview in Kiabana for each system. Infra provisioned via ansible, and our elastic/kibana objects/configs provisioned via terraform, but I would guess this is irrelevant.

We have filebeat sending logs to a datastream with ilm enabled. Datastream called "foo-8.17.1", so backing indexes are like ".ds-foo-8.17.1-2025.02.04-000001".

We also created an index template with aliases (one for each system), via terraform:

resource "elasticstack_elasticsearch_index_template" "foo_index_template" {
  name = "foo-index-template"

  priority       = 200
  index_patterns = ["foo-*"]

  dynamic "template" {
    for_each = var.data_views
    content {
      alias {
        name = template.key
        filter = jsonencode({
          wildcard = {
            "kubernetes.pod.name" = "${template.value}*"
          }
        })
      }
    }
  }
}

What we found is that this only worked when we first created the index template above, and then create the data stream later.
If the datastream is already present, and then we create the index template, we don't get the aliases to the new indexes when we rollover.

Ther same behavior was observed when creating the index template directly via Kibana UI.

Is your Q "what to do now" or "Is the system working correctly?"

The official docs give this sequence:

To set up a data stream, follow these steps:

  1. [Create an index lifecycle policy]
  2. [Create component templates]
  3. [Create an index template]
  4. [Create the data stream]
  5. [Secure the data stream]

The first question is indeed if elastic is working correctly.

The sources below indicate that changes in the index template could be applied to new indexes by rollover, however I wonder why that's not the case for new index templates that matches the datastream/backing indexes.

Official doc.

comment

Thanks for clarifying.

So, I personally don't know much about Terraform, so what were the actual templates produced? i.e. what did ES actually get sent? You can find out by querying the templates via GET _index_template, and look for those Terraform created.

Oh, though it maybe doesn't matter, what in the index template are you changing, after successfully creating the template then data-stream in that order?

My pleasure Keinv =)

Thanks for taking the time. I'm also getting myself familiar with terraform.

About what we are changing in the index template, we are only adding (filtered) aliases, nothing else. The goal is to be able to create data views for each system we need to monitor. The data views point to each respective alias. So each data view should display the logs of a given service.

More information about the big picture:
The index template got created via Terraform without any problems. What seems to have happened is:
We first provisioned filebeat and some other things via ansible. This made filebeat create a default index template and a datastream.

All good, in our test we then created our own index template later, assigning a greater priority. What seems to happen is that this second customized index template is not used. Maybe elastic, during the creation of a data-stream finds what is the matching index template with the highest priority and use that one, and does not look for potential new indexes with even greater priority when rolling over.

We would like to avoid having a strict order when provisioning all those things in a very strict order as from an operational perspective this is not ideal.
For example, we would like that if we first deploy filebeat, elastic will use whatever index template and datastream filebeat created by default, but next, if we deploy newer index templates with higher priority elastic would pick it up.
This behavior happened even when we created things directly in elastic, not via terraform.

This is the first challenge.
The next challenge is that if we use our customized index template, we loose all the mappings that filebeat created automatically, and we would need to add that ourselves in our index template.

So ideally we need to find a way to:

  1. Let filebeat create its index template and datastream
  2. Find a way to sneak filtered aliases in this datastream, hopefully without forcing a strict order in the deployment

But I can't get my head around finding a solution for this.

I think I understand what you are trying to do. Thanks for the fuller context.

I'm just quoting here from the docs for this API

POST /_index_template/{name}

Index templates define settings, mappings, and aliases that can be applied automatically to new indices.

Elasticsearch applies templates to new indices based on an wildcard pattern that matches the index name. Index templates are applied during data stream or index creation. For data streams, these settings and mappings are applied when the stream's backing indices are created. Settings and mappings specified in a create index API request override any settings or mappings specified in an index template. Changes to index templates do not affect existing indices, including the existing backing indices of a data stream.

Last sentence seems relevant here.

Yeah, I've been reading their docs extensively. They clearly mention changes to existing index templates do not affect existing indexes (only the new ones following a rollover).

On my case is: creating new index templates do not get applied to new indexes, following a rollover. Once a datastream and its backing indexes are created and an index template is applied, elastic does not check for newer index templates with higher priority. It doesn't re-assign index templates when rolling over.

I'll guess will have to live with that.

Mmm. I think we head towards effectively just a discussion on the documentation and its semantics. Obviously the software's behavior should match the software's documentation, and vice versa, and I personally dont see an inarguable case here that it doesn't.

The index template you create in step 3 of the process above references the specific data stream you then create in step 4. The data_stream parameter presumably internally and implciitly creates some sort of mapping between that index template and that data stream. The docs say specifically, if you change something in the index template at that point, it won't affect existing backing indices of a data stream.

I agree though it's interesting the create index-template call, for data streams, allow (and the docs have) an example with priority set to something, 500 in the docs. As, as you write, you can later create a very similar looking template with higher priority, but it is not 100% effective, so the "priority" is not completely respected.

I guess you could open some sort of enhancement request to get the behavior you desire. But that would seem to me a possibly breaking change, though that doesn't mean it wouldn't get implemented.

1 Like