Fails to index doc after upgrading to 6.8.1 from 5.6

Elastic Stack Elasticsearch
1

After upgrading my ELK from 5.6 to 6.8 I see index failure in my logstash processing from ala:

[2019-07-06T22:16:55,774][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"collectd-2019.07.06", :_type=>"doc", :routing=>nil}, #<LogStash::Event:0x2c0e7a68>], :response=>{"index"=>{"_index"=>"collectd-2019.07.06", "_type"=>"doc", "_id"=>"hW3uyGsBd4JJZ1FGtS3T", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [host] of type [keyword] in document with id 'hW3uyGsBd4JJZ1FGtS3T'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:57"}}}}}

Assume it some issue with my mapping (not my strongest :slight_smile:

Any hints appreciated, TIA

Mapping looks like below:

{
    "order": 0,
    "index_patterns": [
      "collectd-*"
    ],
    "settings": {
      "index": {
        "mapping.ignore_malformed": true,
        "refresh_interval": "5s"
      }
    },
    "mappings": {
      "_default_" : {
        "dynamic_templates": [
          {
            "strings" : {
              "match_mapping_type" : "string",
              "mapping" : {
                "type" : "text",
                "ignore_above" : 64,
                "norms" : "false",
                "fields" : {
                  "keyword" : {
                    "type" : "keyword",
                    "ignore_above" : 64
                  }
                }
              }
            }
          },
          {
            "not_a_string_type" : {
              "mapping" : {
                "norms" : "false",
                "type" : "{dynamic_type}"
              },
              "match" : "*"
            }
          }
        ],
        "properties": {
          "@timestamp": {
            "type": "date",
            "format": "strict_date_optional_time||epoch_millis"
          },
          "host": {
            "type": "keyword"
          },...

Though I do see quites some docs with data in this field, so are it only some doc's that fails or do docs get indexed afterall?

2

Seemed that both formerly I was using multi type docs in same collectd index which now of course fails, splitting this into a new separate index per sub-type and filebeat 6 also had moved 'host' into '[host][name]' so altering my logstash filters to rename this back to match my mapping fixed this :slight_smile:

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.