Matching patterns from existing older templates ... will take precedence during new index creation

Elastic Stack Elasticsearch
,
1

Hello,

Environment:

  • Elasticsearch 7.12.1 (Basic License)
  • Kibana 7.12.1 (Basic License)
  • Filebeat 7.12.1 (Basic License)
  • Ubuntu 21.04

I have an index template named myapp the index pattern myapp-*.
This index template created by Filebeat. (see details: Legacy index templates)

I'd like to apply 1 ILM policy and 3 templates to every newly created index:

###################################

# https://www.elastic.co/guide/en/elasticsearch/reference/current/set-up-lifecycle-policy.html

curl -X DELETE "http://*MYELKIP*:9200/_ilm/policy/myapp_cleaner_policy"

curl -X PUT "http://*MYELKIP*:9200/_ilm/policy/myapp_cleaner_policy" -H 'Content-Type: application/json' -d'
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "set_priority": {
            "priority": 100
          }
        }
      },
      "delete": {
        "min_age": "5d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}
' | python3 -mjson.tool

curl -X GET "http://*MYELKIP*:9200/_ilm/policy/myapp_cleaner_policy" | python3 -mjson.tool

###################################

# https://www.elastic.co/guide/en/elasticsearch/reference/current/set-up-lifecycle-policy.html#apply-policy-template

curl -X DELETE "http://*MYELKIP*:9200/_index_template/myapp_cleaner_policy_template"

curl -X PUT "http://*MYELKIP*:9200/_index_template/myapp_cleaner_policy_template" -H 'Content-Type: application/json' -d'
{
	"index_patterns": [ "myapp-*" ],
	"priority": 1,
	"template": {
		"settings": {
			"index": {
				"lifecycle": {
					"name": "myapp_cleaner_policy",
					"rollover_alias": ""
				}
			}
		}
	}
}
' | python3 -mjson.tool

curl -X GET "http://*MYELKIP*:9200/_index_template/myapp_cleaner_policy_template" | python3 -mjson.tool

###################################

curl -X DELETE "http://*MYELKIP*:9200/_index_template/myapp_total_fields_policy_template"

curl -X PUT "http://*MYELKIP*:9200/_index_template/myapp_total_fields_policy_template" -H 'Content-Type: application/json' -d'
{
	"priority": 2,
	"index_patterns": [ "myapp-*" ],
	"template": {
		"settings": {
			"index": {
				"mapping": {
					"total_fields": {
						"limit": "5432"
					}
				}
			}
		}
	}
}
' | python3 -mjson.tool

curl -X GET "http://*MYELKIP*:9200/_index_template/myapp_total_fields_policy_template" | python3 -mjson.tool


###################################

curl -X DELETE "http://*MYELKIP*:9200/_index_template/myapp_date_nano_template"

curl -X PUT "http://*MYELKIP*:9200/_index_template/myapp_date_nano_template" -H 'Content-Type: application/json' -d'
{
	"priority": 3,
	"index_patterns": [ "myapp-*" ],
	"template": {
		"mappings": {
			"properties": {
				"@timestamp": {
					"type": "date_nanos"
				},
				"mytimestamp": {
					"type": "date_nanos"
				}
			}
		}
	}
}
' | python3 -mjson.tool

curl -X GET "http://*MYELKIP*:9200/_index_template/myapp_date_nano_template" | python3 -mjson.tool

Short explain:

  • myapp_cleaner_policy: delete an index after 5 days
  • myapp_cleaner_policy_template: apply myapp_cleaner_policy to every newly created index which is match with "myapp-*"
  • myapp_total_fields_policy_template: set the total mapped limit to 5432 to every newly created index which is match with "myapp-*"
  • myapp_date_nano_template: set preconfigured mapping to every newly created index which is match with "myapp-*"

So after I run that script and check the Elasticsearch log, I got this:

[2021-05-23T15:59:41,524][INFO ][o.e.x.i.a.TransportPutLifecycleAction] [x] adding index lifecycle policy [myapp_cleaner_policy]
[2021-05-23T15:59:41,819][INFO ][o.e.c.m.MetadataIndexTemplateService] [x] removing index template [myapp_cleaner_policy_template]
[2021-05-23T15:59:42,092][WARN ][o.e.c.m.MetadataIndexTemplateService] [x] index template [myapp_cleaner_policy_template] has index patterns [myapp-*] matching patterns from existing older templates [myapp] with patterns (myapp => [myapp-*]); this template [myapp_cleaner_policy_template] will take precedence during new index creation
[2021-05-23T15:59:42,096][INFO ][o.e.c.m.MetadataIndexTemplateService] [x] adding index template [myapp_cleaner_policy_template] for index patterns [myapp-*]
[2021-05-23T15:59:42,314][INFO ][o.e.c.m.MetadataIndexTemplateService] [x] removing index template [myapp_total_fields_policy_template]
[2021-05-23T15:59:42,442][WARN ][o.e.c.m.MetadataIndexTemplateService] [x] index template [myapp_total_fields_policy_template] has index patterns [myapp-*] matching patterns from existing older templates [myapp] with patterns (myapp => [myapp-*]); this template [myapp_total_fields_policy_template] will take precedence during new index creation
[2021-05-23T15:59:42,446][INFO ][o.e.c.m.MetadataIndexTemplateService] [x] adding index template [myapp_total_fields_policy_template] for index patterns [myapp-*]
[2021-05-23T15:59:42,665][INFO ][o.e.c.m.MetadataIndexTemplateService] [x] removing index template [myapp_date_nano_template]
[2021-05-23T15:59:42,771][WARN ][o.e.c.m.MetadataIndexTemplateService] [x] index template [myapp_date_nano_template] has index patterns [myapp-*] matching patterns from existing older templates [myapp] with patterns (myapp => [myapp-*]); this template [myapp_date_nano_template] will take precedence during new index creation
[2021-05-23T15:59:42,775][INFO ][o.e.c.m.MetadataIndexTemplateService] [x] adding index template [myapp_date_nano_template] for index patterns [myapp-*]

if a new index created this is in the Elasticsearch log:

[2021-05-23T16:02:17,854][INFO ][o.e.c.m.MetadataDeleteIndexService] [x] [myapp-7.12.1-2021.05.23/1wx3yd-fQ3mNRgojqEN6_w] deleting index
[2021-05-23T16:02:18,183][INFO ][o.e.c.m.MetadataCreateIndexService] [x] [myapp-7.12.1-2021.05.23] creating index, cause [auto(bulk api)], templates [myapp_date_nano_template], shards [1]/[1]
[2021-05-23T16:02:18,533][INFO ][o.e.c.m.MetadataMappingService] [x] [myapp-7.12.1-2021.05.23/xtI4jkMjSluzhOBjrD5ciA] update_mapping [_doc]

So Elasticsearch only applies the last (the largest priority number -> myapp_date_nano_template`) template of the above script.

I expected that all template will be applied to a newly created index.

I've tried:

  • reorder the templates (different priority order) ... but always the latest template win (with largest priority number)
  • "merge" the content of the three templates into one template ... it worked! All the three "rules" applied to the newly created index

Why only the last template has applied to the newly created index?
Is there some kind of limitations?
Is it related to my "legacy index template" question (Legacy index templates) ?

2

Is there a reason you don't want all 3 configs in the same template?

3

Of course, that would be a reasonable response for this situation but I'd like to understand if I missed something or not done on a proper way or it's just an ES bug

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.