Index templates not working in ES 2.4

Elastic Stack Elasticsearch
1

I have the following index templates that have been working up until Elasticsearch 2.3.4. After I upgraded my Elasticsearch to 2.4, it seem to have broken. I am out of idea where I could be doing wrong. Some help is really appreciated.

step-1: I have inserted and verified that the following index template exists:
curl "http://localhost:9200/_template"

{"stats":{"order":0,"template":"stats-*",...,"mappings":{"_default_":{"dynamic_templates":[...{"timestamp_fields":{"match_pattern":"regex","mapping":{"type":"date","doc_values":true},"match":".*[tT]imestamp"}}],"date_detection":false,...}
(full index in Appendix)

step-2: I understand that index templates are only applied during index creation. So I deleted all existing indices first.
curl -XDELETE "http://localhost:9200/stats-*"

Step-3: Then, I insert the data using Bulk API.
curl -XPOST "http://localhost:9200/_bulk" --data-binary @data.json
data.json
{"index":{"_index":"stats-2016.10","_type":"MyType","_id":"xxxxyyyy"}}
{"@version":"1","@timestamp":"2016-10-30T06:00:00Z",....}

Step-4: Then, I checked the field data type (of @timestamp) as follows:
curl -XGET "http://localhost:9200/stats-2016.10/_mapping/MyType/field/@timestamp"

My understanding is that the bulk inserts should cause the indexes to be automatically created and since the index name (stats-2016.10) matches the index template (stats-*), the template should have applied. Then, I expect that fields that end with "[tT]imestamp" should have "date" type mapping. Therefore,

Expected (was seen in ES 2.3.4):

{
"stats-2016.10": {
"mappings": {
"MyType": {
"@timestamp": {
"full_name": "@timestamp",
"mapping": {
"@timestamp": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
}
}
}
}
}
}
}

But, Actual (seen in ES 2.4):

{
"stats-2016.10": {
"mappings": {
"MyType": {
"@timestamp": {
"full_name": "@timestamp",
"mapping": {
"@timestamp": {
"type": "string",
"fields": {
"raw": {
"type": "string",
"index": "not_analyzed"
}
}
}
}
}
}
}
}
}

All the above have been working fine until until ES 2.3.4. I upgrade my ES to version 2.4 and it stops working. Did I miss some configurations that should do but didn't? what else could be wrong? Thanks in advance.

Appendix:
full index template
curl "http://localhost:9200/_template"
{"stats":{"order":0,"template":"stats-*","settings":{"index":{"number_of_shards":"2","number_of_replicas":"0","refresh_interval":"10s"}},"mappings":{"_default_":{"dynamic_templates":[{"string_fields":{"mapping":{"fields":{"raw":{"index":"not_analyzed","omit_norms":true,"type":"string","doc_values":true}}},"match_mapping_type":"string","match":"*","index":"analyzed"}},{"number_fields":{"mapping":{"doc_values":true},"match_mapping_type":"long","match":"*"}},{"boolean_fields":{"mapping":{"doc_values":true},"match_mapping_type":"boolean","match":"*"}},{"timestamp_fields":{"match_pattern":"regex","mapping":{"type":"date","doc_values":true},"match":".*[tT]imestamp"}}],"date_detection":false,"properties":{"@version":{"index":"not_analyzed","type":"string"}}}},"aliases":{}}}

2

Hi @kaung

You should move timestamp_fields to the first in dynamic_templates, I think.
See:
https://www.elastic.co/guide/en/elasticsearch/reference/current/dynamic-templates.html#dynamic-templates

3

@johtani. Thanks and Indeed, swapping the order fixes it. Also since the doc page you mentioned says "Templates are processed in order — the first matching template wins." So that makes sense.

I am curious why it has been working from ES 1.7 thru 2.3.x though. I re-did the test to be sure. Up until ES 2.3.5, the mapping for "[tT]imestamp fields was "date" regardless of the ordering in the index template. It seems to me there used to be something else apart from which template appear first.

4

Did you use exactly same template? Did you set date_detection:false?
I didn't read the code honestly, I just would like to ask it.

5

Yep. It's exactly the same template. date_detection:false. Tested with the same 4 steps listed in my post .