Elasticsearch Mapping (date in index)

Elastic Stack Elasticsearch
1

I am using Flume to put some data into ElasticSearch:

The interceptor is below:
agent.sources.tail.interceptors.i7.serializers = s0
agent.sources.tail.interceptors.i7.serializers.s0.type = org.apache.flume.interceptor.RegexExtractorInterceptorMillisSerializer
agent.sources.tail.interceptors.i7.serializers.s0.name = date
agent.sources.tail.interceptors.i7.serializers.s0.pattern = yyyy MMM dd HH:mm:ss

For some reason the date was coming in as a string even with the above interceptor so decided to do a mapping as shown below:

PUT myindex-2016-03-16{
"mappings": {
"mytype": {
"properties": { "@fields" : {
"properties": {
"date": { "type": "date", "format": "yyyy-MMM-dd HH:mm:ss" } } } } } }}

Flume tries to be clever and creates a new index for each day so the mapping I made above will be invalid for tomorrow as its going to make date appear as a string again on the new index for 17th March, is there a way to tell elastic that any index with myindex* should have the above mapping? As obviously doing a PUT with myindex* and the above does not work.

Thanks

2

You can add a mapping for all indices matching a pattern through the use of a index template.

3

Thanks, I knew elastic had a way but weren't sure what to look for.

thanks again

4

You can create a mapping template for all indices start with myindex

PUT _template/myindex
{
  "template": "myindex*",
  "mappings": {
    "mytype": {
      "properties": {
        "@fields": {
          "properties": {
            "date": {
              "type": "date",
              "format": "yyyy-MMM-dd HH:mm:ss"
            }
          }
        }
      }
    }
  }
}

with "template": "myindex*", your future indices will use the mappings you specify in the template. You can modify (by overwriting) the template at any time, but it affects only future indices.