Hi,
I have struggled with this now for a bit over a week and I just can't seem to figure it out.
I'm running 3 Elastic Stack setups for logging. In one setup I can't get "ignore_malformed": "true" set for some reason. All three setups have been upgraded to 5.4.1 for Filebeat, Logstash, Elasticsearch and Kibana. All running on Debian Jessie (probably slightly different version). All three setups are deployed through Puppet using the same codebase.
Logstash output
cat /etc/logstash/conf.d/output output { if [log_id] == "gcp" { elasticsearch { hosts => ["10.0.255.35:9202", "10.0.255.36:9202", "10.0.255.37:9202", "10.0.255.38:9202"] index => "logstash-gcp-%{+YYYY.MM.dd}" flush_size => 200 template => "/etc/logstash/gcp_template_es_index.json" } } else { elasticsearch { hosts => ["10.0.255.35:9202", "10.0.255.36:9202", "10.0.255.37:9202", "10.0.255.38:9202"] index => "logstash-%{+YYYY.MM.dd}" flush_size => 200 template => "/etc/logstash/default_template_es_index.json" } } }
Template
cat /etc/logstash/gcp_template_es_index.json { "template" : "logstash-gcp-*", "version" : 50001, "settings" : { "index.refresh_interval" : "5s", "index.mapping.ignore_malformed": true }, "mappings" : { "_default_" : { "_all" : {"enabled" : true, "norms" : false}, "dynamic_templates" : [ { "message_field" : { "path_match" : "message", "match_mapping_type" : "string", "mapping" : { "type" : "text", "norms" : false } } }, { "string_fields" : { "match" : "*", "match_mapping_type" : "string", "mapping" : { "type" : "text", "norms" : false, "fields" : { "keyword" : { "type": "keyword" } } } } } ], "properties" : { "@timestamp": { "type": "date", "include_in_all": false }, "@version": { "type": "keyword", "include_in_all": false }, "geoip" : { "dynamic": true, "properties" : { "ip": { "type": "ip" }, "location" : { "type" : "geo_point" }, "latitude" : { "type" : "half_float" }, "longitude" : { "type" : "half_float" } } } } } } }
With GET logstash-2017.06.16/_settings
I see this on the working setup
{ "logstash-2017.06.16": { "settings": { "index": { "mapping": { "ignore_malformed": "true" }, "refresh_interval": "5s", "number_of_shards": "5", "provided_name": "logstash-2017.06.16", "creation_date": "1497571200021", "number_of_replicas": "1", "uuid": "ahGtYCnuQEC_Zs77rrmuqQ", "version": { "created": "5040099" } } } } }
And on the one that I'm struggling with I see this
{ "logstash-gcp-2017.06.16": { "settings": { "index": { "refresh_interval": "5s", "number_of_shards": "5", "provided_name": "logstash-gcp-2017.06.16", "creation_date": "1497571204225", "number_of_replicas": "1", "uuid": "6O9-PEmSS3adnf1pXnbsDQ", "version": { "created": "5040199" } } } } }
So, the mapping
part doesn't seem to get applied for some reason. Logstash has definitely been restarted since my last change to the files. Where should I start looking for the solution? Any help would be greatly appreciated.