Specifying with Elasticsearch template and file to use

Elastic Stack Logstash
1

Hi Guys

This is probably a newbie config mistake.

When specifying a custom Elasticsearch template for an output, it appears the index and template_name need to be the same.. so if I had a generic template that I want to use across several indices, it doesn't work.

So for example if I have the following config

output {
    if [parser] == "fortimail" {
        elasticsearch {
            hosts => localhost
            index => [ "fortimail-%{+YYYY.MM.DD}" ]
            template => "/etc/logstash/templates/fortimail.json"
            template_name => "fortimail-*"
       }
      }
}

Then my fortimail.json looks like this, notice the template name is fortmail-* -

{
  "template" : "fortimail-*",
  "settings" : {
    "index.refresh_interval" : "5s"
  },
  "mappings" : {
    "_default_" : {
       "_all" : {"enabled" : true, "omit_norms" : true},
       "dynamic_templates" : [ {
         "message_field" : {
           "match" : "message",
           "match_mapping_type" : "string",
           "mapping" : {
             "type" : "string", "index" : "analyzed", "omit_norms" : true
           }
         }
       }, {
         "string_fields" : {
           "match" : "*",
           "match_mapping_type" : "string",
           "mapping" : {
             "type" : "string", "index" : "analyzed", "omit_norms" : true,
               "fields" : {
                 "raw" : {"type": "string", "index" : "not_analyzed", "ignore_above" : 256}
               }
           }
         }
       } ],
       "properties" : {
         "@version": { "type": "string", "index": "not_analyzed" },
         "geoip"  : {
           "type" : "object",
             "dynamic": true,
             "properties" : {
               "location" : { "type" : "geo_point" }
             }
         },
         "src_geoip"  : {
           "type" : "object",
             "dynamic": true,
             "properties" : {
               "location" : { "type" : "geo_point" }
             }
         },
         "dst_geoip"  : {
           "type" : "object",
             "dynamic": true,
             "properties" : {
               "location" : { "type" : "geo_point" }
             }
         }
       }
    }
  }
}

This works fine, but if I create a new index by changing the output to

index => [ "MyNewIndex-%{+YYYY.MM.DD}" ]

Then Elasticsearch reverts to using the default Logstash elasticsearch template.. And the only way to get it to work, is to clone the FortiMail.json file and change the the template value to "MyNewIndex-*".

Am I doing something wrong? Or is this how its meant to work?

1 Like
2

Yes, this is how it's supposed to work.

The template pattern ("fortimail-*" in your case) must match the index name. Setting template for an elasticsearch output won't necessarily cause ES to use that template. It'll only cause Logstash to push the template to ES, giving it the name in template_name (which is the name of the template, not the index name pattern it should apply to). If the name of the index Logstash posts data to happens to match your template then it'll be applied. Otherwise not.

2 Likes
3

Thanks Magnus :slight_smile: