Overriding Builtin Datastream Index / Component Templates

Elastic Stack Elasticsearch
#1

Hello,

With legacy index templates, in order to add extra fields without risk that an Elastic update would ovverwrite changes, we created override templates with a higher order which contained extra fields settings etc.

When Elastic would update an index template and it's component templates, I would loose any adjustments I made? Whats the recommended way to add fields or settings which is not influenced by Elastic updates now?

Grtz

Willem

#2

What do you mean by an update would overwrite changes?

#3

@warkolm Thanks for your answer. I mean that the index templates and component templates provided by Elastic could have changed and so could be being overwritten during an update?

#4

For things like system index templates, or modules in Beats etc, yes. For anything outside that, we don't provide index templates, so I am not sure what you are referring to here.

#5

For example if I want to add extra custom fields such as digipolis.team to my data (a field on which we base access controls), then I need to provide a mapping for that. Up to this day we use a higher order template containing those extra custom fields.

So for example with the new templates, I'm wondering on how to do this?

So a simple solution would be to add the extra fields to the component template metrics-mappings:

GET _component_template/metrics-mappings

{
  "component_templates" : [
    {
      "name" : "metrics-mappings",
      "component_template" : {
        "template" : {
          "mappings" : {
            "dynamic_templates" : [
              {
                "strings_as_keyword" : {
                  "mapping" : {
                    "ignore_above" : 1024,
                    "type" : "keyword"
                  },
                  "match_mapping_type" : "string"
                }
              }
            ],
            "date_detection" : false,
            "properties" : {
              "@timestamp" : {
                "type" : "date"
              },
              "ecs" : {
                "properties" : {
                  "version" : {
                    "ignore_above" : 1024,
                    "type" : "keyword"
                  }
                }
              },
              "digipolis" : {
                "properties" : {
                  "team" : {
                    "ignore_above" : 1024,
                    "type" : "keyword"
                  }
                }
              },
              "data_stream" : {
                "properties" : {
                  "namespace" : {
                    "type" : "constant_keyword"
                  },
                  "type" : {
                    "type" : "constant_keyword",
                    "value" : "metrics"
                  },
                  "dataset" : {
                    "type" : "constant_keyword"
                  }
                }
              },
              "host" : {
                "properties" : {
                  "ip" : {
                    "type" : "ip"
                  }
                }
              }
            }
          }
        },
        "version" : 0,
        "_meta" : {
          "managed" : true,
          "description" : "default mappings for the metrics index template installed by x-pack"
        }
      }
    }
  ]
}

digipolis.team would mapped properly in all metrics-* indices. But the metrics-mappings component template might be overwritten during an update? So I'm hoping to find a better option to be able to add extra fields with less risk for issues after updates.

Grtz

Willem

#6

Oh now I see, you're editing the templates for system indices. I am not sure what the best way to approach this is sorry.

#7

@warkolm Could you please ask around, as this is imho very important. All kinds of custom enterprise-specific fields needs to be added all the time and we need a way which guarantees survival of updates, like we currently have with legacy index templates.

Thanks.

#8

We won't provide any assurances here as you're editing system indices. I would suggest raising a feature request on GitHub with your use case and see if there's a future change we can make.

1 Like
#9

Rather than modifying the component template. You might want to try adding a new component template, then assigning it to the metrics index template. I'm unsure if the index template is overridden (but I wouldn't expect it unless you're setting stuff up to explicitly override index templates)

Note: I've never tried this, so not 100% how it would work.

1 Like
#10

@BenB196 Thanks for the suggestion.

You might want to try adding a new component template, then assigning it to the metrics index template

I've been thinking about this (and tested it). Although it would work, it also implies I would have to update the metrics index template. And how sure can I be that this metric template is not updated by Elastic with some update?

For example:

GET _index_template/metrics
{
  "index_templates" : [
    {
      "name" : "metrics",
      "index_template" : {
        "index_patterns" : [
          "metrics-*-*"
        ],
        "composed_of" : [
          "metrics-mappings",
          "metrics-settings",
          "mycustomfieldmappings",
          "mycustomindexsettings"
        ],
        "priority" : 100,
        "version" : 0,
        "_meta" : {
          "description" : "default metrics template installed by x-pack",
          "managed" : true
        },
        "data_stream" : { }
      }
    }
  ]
}

For example if I add my custom field in the mycustomfieldmappings component template, I'd have to add it to the above index template provided by Elastic. But if Elastic would at some point update this metrics index template my addition of the mycustomfieldmappings component template (and for example mycustomindexsettings) would be gone?

#11

If you want to guarantee that your change will never be overwritten. The solution would be create a new Index Template (lets call it metrics_additions). That index template would use the same index pattern as the metrics template (metrics-*-*). Assign your custom component templates to it, then when any new metrics-*-* indexes are created, they should get both the metrics mappings and the metrics_additions mappings.

Edit: One caveat with this solution is mapping conflicts can occur so you'll want to make sure priorities are set properly on the index templates.

#12

@BenB196 Thanks again for the idea. I'm not sure it would work though, as the documentation (https://www.elastic.co/guide/en/elasticsearch/reference/current/index-templates.html)

says:

If a new data stream or index matches more than one index template, the index template with the highest priority is used.

So doesn't this imply only one index_template (and it's component templates) will match?

(Mapping conflicts should not be an issue as the main thing I want to do here is add a custom object very specific for our organization)

#13

As a reference, this is the object I need to add to 'all' of our data:

"digipolis": {
  "properties": {
    "application": {
      "properties": {
        "id": {
          "ignore_above": 1024,
          "type": "keyword"
        },
        "name": {
          "ignore_above": 1024,
          "type": "keyword"
        },
        "type": {
          "ignore_above": 1024,
          "type": "keyword"
        }
      }
    },
    "correlation_id": {
      "ignore_above": 1024,
      "type": "keyword"
    },
    "environment": {
      "ignore_above": 1024,
      "type": "keyword"
    },
    "subcel": {
      "ignore_above": 1024,
      "type": "keyword"
    }
  }
}
#14

Based on these warnings in the docs it should in theory work (I believe this is how the sub-index templates work; ex: metrics-*-* technically matches metrics-system.core-*) You'd need to test it out though:

Elasticsearch has built-in index templates for the metrics-*-* , logs-*-* , and synthetics-*-* index patterns, each with a priority of 100 . Elastic Agent uses these templates to create data streams. If you use Elastic Agent, assign your index templates a priority lower than 100 to avoid an overriding the built-in templates.

Otherwise, to avoid accidentally applying the built-in templates, use a non-overlapping index pattern or assign templates with an overlapping pattern a priority higher than 100 .

For example, if you don’t use Elastic Agent and want to create a template for the logs-* index pattern, assign your template a priority of 200 . This ensures your template is applied instead of the built-in template for logs-*-* .

#15

Ok, thanks @BenB196, I must say I have doubts this will work with a metrics_additions template which would combine the settings and mappings from the builtin metrics template with the metrics_additions template (like we can do now with legacy templates).

What will work is create a metrics_additions template which has a higher priority, but then only the settings from the metrics_additions would applied (I assume based on what the docs say). This would have other implications as this would mean my final mappings might miss some stuff provided by Elastic..

Anyway I guess I'll need to test it out...

Do you know when the legacy templates will be fully deprecated? Will this be in 8 already? I really hope not, because it will be an awful lot of work to migrate everything to datastreams :frowning:

#16

I don't know when they'll officially be deprecated. However, the new index templates don't require you to use datastreams (it's checkbox). So really you can do some minor conversion from legacy to component, without changing the underlying index structure.

1 Like