Hi,
I have installed ELK (eleasticsearch 2.3.1 logstash 2.3.0 and kibana 4.5). when the system is started and ELK is started for the first time (via linux services).
Logstash mappings are not created with my own template (missing e.g. the .raw files).
In the elastics search logging i get the following logging:
[2016-05-30 10:07:01,794][INFO ][cluster.metadata ] [node-1] [my-logstash-2016.05.28] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings [sysmon_x]
[2016-05-30 10:07:01,921][INFO ][cluster.metadata ] [node-1] [my-logstash-2016.05.29] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings [messages]
Reason i use this template because sometime the raw fields are not created at startup.
I want to use my own template to have more control over the fields (which will have .raw fields and which don't).
My Question is, why is my template not used?
---------- EXTRA INFO BELOW ----------
Logstash output.conf:
output{
if "_grokparsefailure" in [tags]{
#stdout { codec => rubydebug }
file {
codec => rubydebug
path => "/tmp/logstash.log"
}
}
elasticsearch {
hosts => "my-1:13076"
index => "my-logstash-%{+YYYY.MM.dd}"
template => "/etc/logstash/template/elasticsearch-template.json"
template_name => "my-logstash*"
template_overwrite => "true"
manage_template => "false"
flush_size => 100
workers => 1
}
}
My template:
90% same as default.
different all enabled = false
and removed ignore above 256 for raw field
{
"template" : "my-logstash*",
"settings" : {
"index.refresh_interval" : "5s"
},
"mappings" : {
"_default_" : {
"_all" : {"enabled" : false, "omit_norms" : true},
"dynamic_templates" : [ {
"message_field" : {
"match" : "message",
"match_mapping_type" : "string",
"mapping" : {
"type" : "string", "index" : "analyzed", "omit_norms" : true,
"fielddata" : { "format" : "disabled" }
}
}
}, {
"string_fields" : {
"match" : "*",
"match_mapping_type" : "string",
"mapping" : {
"type" : "string", "index" : "analyzed", "omit_norms" : true,
"fielddata" : { "format" : "disabled" },
"fields" : {
"raw" : {"type": "string", "index" : "not_analyzed", "doc_values" : true}
}
}
}
}, {
"float_fields" : {
"match" : "*",
"match_mapping_type" : "float",
"mapping" : { "type" : "float", "doc_values" : true }
}
}, {
"double_fields" : {
"match" : "*",
"match_mapping_type" : "double",
"mapping" : { "type" : "double", "doc_values" : true }
}
}, {
"byte_fields" : {
"match" : "*",
"match_mapping_type" : "byte",
"mapping" : { "type" : "byte", "doc_values" : true }
}
}, {
"short_fields" : {
"match" : "*",
"match_mapping_type" : "short",
"mapping" : { "type" : "short", "doc_values" : true }
}
}, {
"integer_fields" : {
"match" : "*",
"match_mapping_type" : "integer",
"mapping" : { "type" : "integer", "doc_values" : true }
}
}, {
"long_fields" : {
"match" : "*",
"match_mapping_type" : "long",
"mapping" : { "type" : "long", "doc_values" : true }
}
}, {
"date_fields" : {
"match" : "*",
"match_mapping_type" : "date",
"mapping" : { "type" : "date", "doc_values" : true }
}
}, {
"geo_point_fields" : {
"match" : "*",
"match_mapping_type" : "geo_point",
"mapping" : { "type" : "geo_point", "doc_values" : true }
}
} ],
"properties" : {
"@timestamp": { "type": "date", "doc_values" : true },
"@version": { "type": "string", "index": "not_analyzed", "doc_values" : true },
"geoip" : {
"type" : "object",
"dynamic": true,
"properties" : {
"ip": { "type": "ip", "doc_values" : true },
"location" : { "type" : "geo_point", "doc_values" : true },
"latitude" : { "type" : "float", "doc_values" : true },
"longitude" : { "type" : "float", "doc_values" : true }
}
}
}
}
}
}