Failure when parsing logs -> results in a lot of worthless Logs

Elastic Stack Logstash
1

Hello community -

i've set up a logstash config to parse some VmWare Host Logs, but i see an error in my Logstash logs:

error:

:response=>{"index"=>{"_index"=>"vmware-2020.05.14", "_type"=>"_doc", "_id"=>"ejurEnIBcPJpTta85B7J", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse field [host] of type [text] in document with id 'ejurEnIBcPJpTta85B7J'. Preview of field's value: '{name=MTXSWSQL01.matrix.int}'", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:871"}}}}}

Here's my Config for these kind of logs:

input {
        udp {
        port => 5140
                type => syslog
                }
        }
filter {
if [message] =~ /-esx-/ {
 if [message] =~ /^\S+ \S+ \S+ \S+: \S+ \S+ \[Originator@\d+ [^]]+\] .*$/
    {
      grok {
        match => [ "message", "\S+ \S+ (?<syslog_hostname>\S+) (?<esxservice>\S+): (?<level>\S+) (?<esxprocess>\S+) \[Originator@\d+ (?<esxsubinfo>[^]]+)\] (?<esxmessage>.*)" ]
      }
      kv {
        source=>"esxsubinfo"
      }
    }
    else {
      grok {
        match => [ "message", "^\S+ \S+ (?<syslog_hostname>\S+) (?<esxservice>\S+): (?<esxservicemessage>.*)$" ]
      }
    }
}
}
 output {
      elasticsearch {
        hosts => ["X.X.X.X:9200"]
        #index => "vmware"
        index => "vmware-%{+YYYY.MM.dd}"
        user => 
        password => 
        cacert => '/etc/logstash/certs/ca.crt'
        ssl_certificate_verification => false

  }
}

What do i have to fix? I already rechecked the old config - but cannot spot my error.

Thanks in advance! :slight_smile:

2

This happens when there is a conflict in the datatype of the incoming event and the datatype defined in the mappings in Elasticsearch.

Find out what the defined datatype for that field in Elasticsearch is by doing GET /vmware-2020.05.14/_mapping and change the field datatype in your log to match the datatype set in the mapping. If you want to avoid this problem going forward , make sure to update your Logstash parse code so that the field is correctly parsed before getting indexed.

3

Hi Kumar!

Thanks for the fast reply -

that's what i'm aiming at - fixing the parse code.

But as far as i can tell - this field is parsed by the KV Filter Plugin - how can i change that? I do Not parse the field [host] in my RegEX as you can see above.

4

the host field is automatically generated by logstash, and it defaults to , i think, the hostname where you run logstash. you can see this field being generated if you set output to stdout.

if you want the field to be indexed in ES, what’s the data type for host in elasticsearch mapping? they need to match.

if you don’t need them, just drop the host field

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.