Initial startup (bulk api) does not use my template

Elastic Stack Logstash
1

Hi,

I have installed ELK (eleasticsearch 2.3.1 logstash 2.3.0 and kibana 4.5). when the system is started and ELK is started for the first time (via linux services).
Logstash mappings are not created with my own template (missing e.g. the .raw files).

In the elastics search logging i get the following logging:

[2016-05-30 10:07:01,794][INFO ][cluster.metadata         ] [node-1] [my-logstash-2016.05.28] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings [sysmon_x]
[2016-05-30 10:07:01,921][INFO ][cluster.metadata         ] [node-1] [my-logstash-2016.05.29] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings [messages]

Reason i use this template because sometime the raw fields are not created at startup.
I want to use my own template to have more control over the fields (which will have .raw fields and which don't).

My Question is, why is my template not used?

---------- EXTRA INFO BELOW ----------

Logstash output.conf:

output{
        if "_grokparsefailure" in [tags]{
                #stdout { codec => rubydebug }
                file {
                        codec => rubydebug
                        path => "/tmp/logstash.log"
                }
        }
        elasticsearch { 
                hosts => "my-1:13076"
                index => "my-logstash-%{+YYYY.MM.dd}"
                template => "/etc/logstash/template/elasticsearch-template.json"
                template_name => "my-logstash*"
                template_overwrite => "true"
                manage_template => "false"
                flush_size => 100
                workers => 1
        }
}

My template:
90% same as default.
different all enabled = false
and removed ignore above 256 for raw field

{
  "template" : "my-logstash*",
  "settings" : {
    "index.refresh_interval" : "5s"
  },
  "mappings" : {
    "_default_" : {
      "_all" : {"enabled" : false, "omit_norms" : true},
      "dynamic_templates" : [ {
        "message_field" : {
          "match" : "message",
          "match_mapping_type" : "string",
          "mapping" : {
            "type" : "string", "index" : "analyzed", "omit_norms" : true,
            "fielddata" : { "format" : "disabled" }
          }
        }
      }, {
        "string_fields" : {
          "match" : "*",
          "match_mapping_type" : "string",
          "mapping" : {
            "type" : "string", "index" : "analyzed", "omit_norms" : true,
            "fielddata" : { "format" : "disabled" },
            "fields" : {
              "raw" : {"type": "string", "index" : "not_analyzed", "doc_values" : true}
            }
          }
        }
      }, {
        "float_fields" : {
          "match" : "*",
          "match_mapping_type" : "float",
          "mapping" : { "type" : "float", "doc_values" : true }
        }
      }, {
        "double_fields" : {
          "match" : "*",
          "match_mapping_type" : "double",
          "mapping" : { "type" : "double", "doc_values" : true }
        }
      }, {
        "byte_fields" : {
          "match" : "*",
          "match_mapping_type" : "byte",
          "mapping" : { "type" : "byte", "doc_values" : true }
        }
      }, {
        "short_fields" : {
          "match" : "*",
          "match_mapping_type" : "short",
          "mapping" : { "type" : "short", "doc_values" : true }
        }
      }, {
        "integer_fields" : {
          "match" : "*",
          "match_mapping_type" : "integer",
          "mapping" : { "type" : "integer", "doc_values" : true }
        }
      }, {
        "long_fields" : {
          "match" : "*",
          "match_mapping_type" : "long",
          "mapping" : { "type" : "long", "doc_values" : true }
        }
      }, {
        "date_fields" : {
          "match" : "*",
          "match_mapping_type" : "date",
          "mapping" : { "type" : "date", "doc_values" : true }
        }
      }, {
        "geo_point_fields" : {
          "match" : "*",
          "match_mapping_type" : "geo_point",
          "mapping" : { "type" : "geo_point", "doc_values" : true }
        }
      } ],
      "properties" : {
        "@timestamp": { "type": "date", "doc_values" : true },
        "@version": { "type": "string", "index": "not_analyzed", "doc_values" : true },
        "geoip"  : {
          "type" : "object",
          "dynamic": true,
          "properties" : {
            "ip": { "type": "ip", "doc_values" : true },
            "location" : { "type" : "geo_point", "doc_values" : true },
            "latitude" : { "type" : "float", "doc_values" : true },
            "longitude" : { "type" : "float", "doc_values" : true }
          }
        }
      }
    }
  }
}
2

Sometimes i get the following (might it have todo with starting up sequence?):

[2016-05-30 12:57:41,722][INFO ][cluster.metadata         ] [node-1] [my-logstash-2016.05.30] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings [karaf, ospl, asl, sysmon_x]
[2016-05-30 12:57:42,210][INFO ][cluster.metadata         ] [node-1] [my-logstash-2016.05.29] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings [messages]
[2016-05-30 12:57:42,652][INFO ][cluster.metadata         ] [node-1] [my-logstash-2016.05.26] creating index, cause [auto(bulk api)], templates [my-logstash*], shards [5]/[1], mappings [_default_, local3]
[2016-05-30 12:57:42,728][INFO ][cluster.metadata         ] [node-1] [my-logstash-2016.05.28] creating index, cause [auto(bulk api)], templates [my-logstash*], shards [5]/[1], mappings [_default_, sysmon_x]

this is executed with the following output.conf of logstash:

output{
        if "_grokparsefailure" in [tags]{
                #stdout { codec => rubydebug }
                file {
                        codec => rubydebug
                        path => "/tmp/logstash.log"
                }
        }
        elasticsearch { 
                hosts => "my-1:13076"
                index => "my-logstash-%{+YYYY.MM.dd}"
                template => "/etc/logstash/template/elasticsearch-template.json"
                template_name => "my-logstash*"
                template_overwrite => "true"
                flush_size => 100
                workers => 1
        }
}
3

Do you have other templates in ES?

4

Hi mark,

No i don't have any other costum templates in es. Only the default es might use for example for kibana.

5

After some testing i see:

  • When i start Elasticsearch, Logstah, Kibana manual then reboot the system => OK
  • When i reboot first (so first time start is done by the system) => NOK

Both situation it is started by root.

any suggestions?

6

Hi,

I see the same result now for topbeat also. Some times the template is not used to created the mappings (at first start).

Could help overwrite template setting?
Or doesn't that work when the default template is used?

Or is there a way to inject the templates / mappings manually?

7

Hi, i got the same problem, do you have solved it?

8

No I still have the issue. For me the workaround works.
Workaround: start logstash/topbeat/metricbeat/kibana on the system where elasicsearch is running first. After that start on the other systems logstash/topbeat/metricbeat.

Hope it helps.