Dissector mapping, field found in event but it was empty

[Badger]

Dissect does not like this user's useragent string. All the fields get parsed correctly, including useragent, but it still gets tagged with _dissectfailure.

[2017-11-16T15:28:04,752][WARN ][org.logstash.dissect.Dissector] Dissector mapping, field found in event but it was empty {"field"=>"message", "event"=>{"web.method"=>"GET", "web.cbytes"=>"226", "web.sbytes"=>"294", "web.port"=>"8081", "web.timetaken"=>"62", "message"=>"2017-11-01 13:10:53 1.2.3.4 GET /favicon.ico - 8081 - 5.6.7.8 QQæµè§å¨+7.9.0+(iPhone;+iOS+11.0.3;+zh_CN) - 302 0 294 226 62", "web.referer"=>"-", "web.subresponse"=>"0", "web.useragent"=>"QQæµè§å¨+7.9.0+(iPhone;+iOS+11.0.3;+zh_CN)", "web.server_ip"=>"1.2.3.4", "@timestamp"=>2017-11-16T20:28:04.732Z, "web.request"=>"/favicon.ico", "@version"=>"1", "host"=>"sd-0364-9b52.nam.nsroot.net", "web.user_id"=>"-", "web.sysdate"=>"2017-11-01 13:10:53", "web.querystring"=>"-", "web.responsestatus"=>"302", "web.client_ip"=>"5.6.7.8"}}

The filter is

filter {
    dissect {
        mapping => { "message" => "%{web.sysdate} %{+web.sysdate} %{web.server_ip} %{web.method} %{web.request} %{web.querystring} %{web.port} %{web.user_id} %{web.client_ip} %{web.useragent} %{web.referer} %{web.responsestatus} %{web.subresponse} %{web.sbytes} %{web.cbytes} %{web.timetaken}" }
   }
}

According to 'od -ha' the user agent string starts with

0000000    5151    b5e6    e88f    88a7    99e5    2ba8    2e37    2e39
          Q   Q   f   5  si   h   '  bs   e  em   (   +   7   .   9   .

[michaelcheung]

I have similar issue also. I updated my docker base image from logstash 5.6.1 to 6.0.0. Using same input message and dissect filter, in 6.0.0, I get _dissectfailure in tags array. But other fields are parsed correctly. This is a slight concern as I would check any failure from tags afterwards.

[michaelcheung]

I have further tested this issue. I guessed it may be related to string length checking in dissect. The line I suspect: https://github.com/logstash-plugins/logstash-filter-dissect/blob/v1.1.2/src/main/java/org/logstash/dissect/JavaDissectorLibrary.java#L187

I reproduce with following case, using utf8 chinese character.

dissect {
    mapping => { message => "%{name}" }
}

And message is simply 子

[Eugene_Aikashev]

I have similar issue too. Any ideas?

[2017-12-01T17:08:47,940][WARN ][org.logstash.dissect.Dissector] Dissector mapping, field found in event but it was empty {"field"=>"message", "event"=>{"@version"=>"1", "host"=>"x.x.x.x", "@timestamp"=>2017-12-01T15:08:47.935Z, "message"=>"{"event_type":"FirewallAggregated_Event","ipv4":"x.x.x.x","hostname":"xxxx","source_uuid":"94d52e7b-680f-47a9-a82b-c87e3ff43e4d","occured":"01-Dec-2017 15:01:09","severity":"Warning","event":"Detected Port Scanning attack","source_address":"xxxx","source_address_type":"IPv4","source_port":63966,"target_address":"xxxx","target_address_type":"IPv4","target_port":1723,"protocol":"TCP","inbound":true,"aggregate_count":1}\n", "type"=>"eset"}}

[Robert_Schultz]

Same here.

dissect {
  mapping => { "message" => "%{prefix}: %{kvp}" }
}

The message is
Info FIREWALL01 Remove: type=FWD|proto=TCP|srcIF=bond1.1|...
with [...] being key value pairs separated by "|" and lasting for 23 more pairs with a length of 346 characters, codec is charset => "ISO8859-1" on the input.
Message starts with a whitespace, this is not reproducible here somehow...

Seems to have appeared with the change from 6.0.0-rc to 6.0.0
Since then the "Info FIREWALL01 Remove:" could also not be dissected properly with "%{severity} %{name} %{action}:"

Any hints?

[guyboertje]

This works for me. Note the three dissections. This can be combined into one though.

input {
  generator {
    lines => [
      " Info FIREWALL01 Remove: type=FWD|proto=TCP|srcIF=bond1.1"
    ]
    count => 1
  }
}

filter {
  dissect {
    mapping => {
      message => " %{prefix}: %{kvp}"
      prefix => "%{severity} %{name} %{action}"
      kvp => "%{?k1}=%{&k1}|%{?k2}=%{&k2}|%{?k3}=%{&k3}"
    }
  }
}

output {
  stdout {
    codec => rubydebug
  }
}

Result:

{
      "sequence" => 0,
          "type" => "FWD",
           "kvp" => "type=FWD|proto=TCP|srcIF=bond1.1",
         "srcIF" => "bond1.1",
      "severity" => "Info",
    "@timestamp" => 2017-12-20T16:30:21.594Z,
          "host" => "Elastics-MacBook-Pro.local",
        "prefix" => "Info FIREWALL01 Remove",
        "action" => "Remove",
       "message" => " Info FIREWALL01 Remove: type=FWD|proto=TCP|srcIF=bond1.1",
      "@version" => "1",
         "proto" => "TCP",
          "name" => "FIREWALL01"
}