Settings in Logstash output config file for "template =>" not applied

Hi,
I have struggled with this now for a bit over a week and I just can't seem to figure it out.

I'm running 3 Elastic Stack setups for logging. In one setup I can't get "ignore_malformed": "true" set for some reason. All three setups have been upgraded to 5.4.1 for Filebeat, Logstash, Elasticsearch and Kibana. All running on Debian Jessie (probably slightly different version). All three setups are deployed through Puppet using the same codebase.

Logstash output

cat /etc/logstash/conf.d/output
output {
  if [log_id] == "gcp" {
    elasticsearch {
      hosts => ["10.0.255.35:9202", "10.0.255.36:9202", "10.0.255.37:9202", "10.0.255.38:9202"]
      index => "logstash-gcp-%{+YYYY.MM.dd}"
      flush_size => 200
      template => "/etc/logstash/gcp_template_es_index.json"
    }
  }
  else {
    elasticsearch {
      hosts => ["10.0.255.35:9202", "10.0.255.36:9202", "10.0.255.37:9202", "10.0.255.38:9202"]
      index => "logstash-%{+YYYY.MM.dd}"
      flush_size => 200
      template => "/etc/logstash/default_template_es_index.json"
    }
  }
}

Template

cat /etc/logstash/gcp_template_es_index.json
{
  "template" : "logstash-gcp-*",
  "version" : 50001,
  "settings" : {
    "index.refresh_interval" : "5s",
    "index.mapping.ignore_malformed": true
  },
  "mappings" : {
    "_default_" : {
      "_all" : {"enabled" : true, "norms" : false},
      "dynamic_templates" : [ {
        "message_field" : {
          "path_match" : "message",
          "match_mapping_type" : "string",
          "mapping" : {
            "type" : "text",
            "norms" : false
          }
        }
      }, {
        "string_fields" : {
          "match" : "*",
          "match_mapping_type" : "string",
          "mapping" : {
            "type" : "text", "norms" : false,
            "fields" : {
              "keyword" : { "type": "keyword" }
            }
          }
        }
      } ],
      "properties" : {
        "@timestamp": { "type": "date", "include_in_all": false },
        "@version": { "type": "keyword", "include_in_all": false },
        "geoip"  : {
          "dynamic": true,
          "properties" : {
            "ip": { "type": "ip" },
            "location" : { "type" : "geo_point" },
            "latitude" : { "type" : "half_float" },
            "longitude" : { "type" : "half_float" }
          }
        }
      }
    }
  }
}

With GET logstash-2017.06.16/_settings I see this on the working setup

{
  "logstash-2017.06.16": {
    "settings": {
      "index": {
        "mapping": {
          "ignore_malformed": "true"
        },
        "refresh_interval": "5s",
        "number_of_shards": "5",
        "provided_name": "logstash-2017.06.16",
        "creation_date": "1497571200021",
        "number_of_replicas": "1",
        "uuid": "ahGtYCnuQEC_Zs77rrmuqQ",
        "version": {
          "created": "5040099"
        }
      }
    }
  }
}

And on the one that I'm struggling with I see this

{
  "logstash-gcp-2017.06.16": {
    "settings": {
      "index": {
        "refresh_interval": "5s",
        "number_of_shards": "5",
        "provided_name": "logstash-gcp-2017.06.16",
        "creation_date": "1497571204225",
        "number_of_replicas": "1",
        "uuid": "6O9-PEmSS3adnf1pXnbsDQ",
        "version": {
          "created": "5040199"
        }
      }
    }
  }
}

So, the mapping part doesn't seem to get applied for some reason. Logstash has definitely been restarted since my last change to the files. Where should I start looking for the solution? Any help would be greatly appreciated.

Was logstash-gcp-2017.06.16 created after the template was updated? What if you create a brand new index yourself, say logstash-gcp-2017.06.30, and inspects its mappings?

Wait. The /_settings API retrieves the index settings. I don't think the index mappings are ever included in that response. Try the /_mappings API instead.

Hi Magnus,
thanks for the reply, much appreciated :slight_smile:

Yes, the indices I show the _settings for both where created after the last change to the Logstash output config and the JSON template files.

I had the same idea of creating an index to see what happens so I did this

PUT logstash-2017.06.20
{
  "settings": {
    "index.mapping.ignore_malformed": true 
  }
}

which means there will be an index ready for todays logs with "index.mapping.ignore_malformed": truealready set. Now I have to wait one more day to see if the next one will have this setting without intervention...

Because I'm setting "index.mapping.ignore_malformed": true for the whole index it shows up under _settings and not _mappings, as far as I can tell based on this ignore_malformed | Elasticsearch Guide [8.11] | Elastic

Cheers,
AB

Quick update: The next daily index that Logstash created is again without "index.mapping.ignore_malformed": true

So, creating the index and applying the setting from Kibana works but using a JSON template in Logstash does not, in this one case. On two other setups that should be identical (as they use the same version of the whole stack and same puppet code base) this works...

Any suggestions on what I could try next are much appreciated...

I solved this by creating a cron job that creates the daily indices before logstash does it.

#!/bin/bash

DATE=$(date --date='1 day' +%Y.%m.%d)

curl -XPUT "http://logs.foo.bar:9200/logstash-$DATE" -d'
{
  "settings": {
    "index.mapping.ignore_malformed": true
  }
}'

In case that helps someone. Not the most elegant solution but looks like it's doing what I need...

AB

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.