Templates for Elastic Search

Elastic Stack Logstash
1

I have a full configuration running (logstash/elastic search/kibana)

As I would like to set dynamic fields as not_analyzed, I created a template on logstash side (with a name matching the index name used in elastic search output section of logstash)

As for now, the fields are still analyzed.

I am a little bit puzzled by the template management, do I need to upload it also on the elastic search side ?

{
  "template" : ""elk-xyz-logs-*",
  "settings" : {
    "index.refresh_interval" : "5s"
  },
  "mappings" : {
    "_default_" : {
      "_all" : {"enabled" : true, "omit_norms" : true},
      "dynamic_templates" : [ {
        "message_field" : {
          "match" : "message",
          "match_mapping_type" : "string",
          "mapping" : {
            "type" : "string", "index" : "analyzed", "omit_norms" : true,
            "fielddata" : { "format" : "disabled" }
          }
        }
      }, {
        "string_fields" : {
          "match" : "*",
          "match_mapping_type" : "string",
          "mapping" : {
            "type" : "string", "index" : "not_analyzed", "omit_norms" : true
          }
        }
      }, {

output {
 file {
  path => "/var/log/logstash/output.log"
 }
 elasticsearch {
  index => "elk-xyz-logs-%{+YYYY.MM.dd}"
  template => "/etc/logstash/templates/elk-xyz-logstash.json"
  template_overwrite => true
  manage_template => false
 }
}

Side note:
My first attempt was to create my template file in conf.d directory => that is a bad idea as it looks like all the files in this directory are merged when logstash starts...

2

Templates are nothing to do with Logstash, they are only for the Elasticsearch side. More info here.

Kopf (an Elasticsearch admin tool plugin) has a nice Index Template editor.

3

I understand that the templates are only useful on the ES side but using the elastic search plugin output, they are kind of managed by logstash (or at least the plugin as it looks like you can define a template file and it will be uploaded automatically to ES )

4

Ah sorry, misread your message. I've never used that feature myself. Have you verified that Elasticsearch has your template stored?

5

You're disabling manage_template so Logstash won't upload the template you're pointing it at. Either enable manage_template again or take over the template ownership by uploading the template yourself outside of Logstash.

6

Thanks to your information, I modified the configuration file.

My first try was still failing but enabling verbose output in logstash pointed me to the right direction, during startup a parsing error was raised during the import of my template file. I fixed the typo and made some other tests and curiously sometimes it was working sometimes not.
I was using a host array in my logstash elasticsearch output config and 2 my nodes are currently turned off, so some of the requests were failing (I initially thought the array was there for fail-over but I RTFM and it's more for load-balancing ^^), listing only up&running hosts was the good configuration, the template is correctly uploaded (and listed using the curl -XGET http://:/_template command)

So , thanks a lot for your quick feedback.