Fails to index due to conflicting doc type

Elastic Stack Logstash
1

Got the classic issue with a 6.8.3 cluster, I defined an index template and created it with ?include_type_name=false

{
  "index_patterns": [ "esxi_firewall-*" ],
  "mappings": {
    "properties": {
      "@timestamp": {
        "type": "date"
      },
      "fw": {
        "properties": {
          "ttl" : { "type": "long" },
          "len" : { "type": "long" },
          "id" : { "type": "long" },
          "urgp" : { "type": "long" },
          "window" : { "type": "long" },
          "tos" : { "type": "keyword" },
          "res" : { "type": "keyword" },
          "prec" : { "type": "keyword" },
          "proto" : { "type": "keyword" },
          "mark" : { "type": "keyword" },
          "out" : { "type": "keyword" }
        }
      },
      "ecs": {
        "properties": {
          "version": {
            "type": "keyword"
          }
        }
      },
      "message": {
        "type": "text",
        "fields": {
           "keyword": { "type" : "keyword" }
        }
      },
      "host": {
        "properties": {
          "name": {
            "type": "keyword"
          }
        }
      },
      "destination": {
        "properties": {
          "address": {
            "type": "keyword",
            "fields": {
              "text": { "type" : "text" }
            }
          },
          "ip": {
            "type": "ip"
          },
          "port": {
            "type": "long"
          }
        }
      },
      "source": {
        "properties": {
          "address": {
            "type": "keyword",
            "fields": {
              "text": { "type" : "text" }
            }
          },
          "ip": {
            "type": "ip"
          },
          "port": {
            "type": "long"
          }
        }
      }
    }
  }
}

but when logstash attempt to index documents I get the conflicting document type:

[2019-09-23T21:59:42,025][WARN ][logstash.outputs.elasticsearch] Could not index
sponse=>{"index"=>{"_index"=>"esxi_firewall-2019.09.23", "_type"=>"doc", "_id"=>...
would have more than 1 type: [_doc, doc]"}}}}

output plugin configured without any document_type:

elasticsearch {
   #cluster => 'mxes2data'
   id => 'mxes2data'
   index => '%{[@metadata][esindex]}-%{+YYYY.MM.dd}'
   action => 'index'
   codec => 'plain'
   user => '<redacted>'
   password => '<redacted>'
   sniffing => false
   manage_template => false
   template_overwrite => true
   hosts => ['<redacted>:9200','<redacted>:9200','<redacted>:9200']
}

}

wondering from where the 'doc' value comes?

Also attempting to map as much as possible to ECS 1.1, only not sure if the core field: ecs.version is just a dotted name or true nested and what nested field prefix to hide custom field under.

Any hints appreciated!

2

An old version of filebeat?

3

Nope got no data or no index until first doc/event is processed by logstash. It'll create the index but it holds no documents after this and logstash just complains as if it is trying to store the first 'doc' typed document, only ES refuses as it expects _type to be _doc...

4

Right, and the first document that is indexed has type _doc, so you cannot subsequently insert a document of type doc.

5

But the first doc to get indexed is failing as such and I don’t specify any document type anywhere, so from where is the type doc picked up by logstash?

6

It looks like doc and not _doc is the default of the elasticsearch output plugin. Is this really correct??

7

Seems that way as tcpdump shows this is POSTed from logstash 6.8.3 for the first doc and ES expects type: _doc:

..[.....POST /_bulk HTTP/1.1
Connection: Keep-Alive
Content-Type: application/json
Content-Length: 942
Host: <redacted>:9200
User-Agent: Manticore 0.6.4
Accept-Encoding: gzip,deflate
Authorization: Basic <redacted>

{"index":{"_id":null,"_index":"esxi_firewall-2019.09.24","_type":"doc","routing":null}}
....

And are not allowed to do mutate+add_field for meta data field _type, so howto alter this I dunno :confused:

8

Changed the template to include the type 'doc' for now... to get this to work :slight_smile:

1 Like
9

Any comments concerning ECS adaption?

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.