Kv plugin not parsing null value

Elastic Stack Logstash
1

using kv to parse iptables logs, i cant get it to save the MAC field

[2928868.448110] [iptables] LOGDROP IN=eth0 OUT= MAC=00:50:56:9a:13:2a:02:1f:a0:00:0d:01:08:00 SRC=5.188.207.7 DST=8.8.8.8 LEN=60 TOS=0x08 PREC=0x20 TTL=50 ID=36130 DF PROTO=TCP SPT=31514 DPT=993 WINDOW=29200 RES=0x00 SYN URGP=0

the problem clearly is in the OUT field, which doesn't have a value, so i end up with

"OUT": "MAC=00:50:56:9a:13:2a:02:1f:a0:00:0d:01:08:00",

if i explicitly declare the fields to be parsed with include_keys and recursive

kv {
    add_tag                 => [ "iptables", "logdrop"]
    allow_duplicate_values  => false
    include_keys            => [ "DPT", "DST", "MAC", "PROTO", "SPT", "SRC" ]
    recursive               => true
}

i don't get a OUT field at all (that's ok), but no MAC field as well (not ok)

expected result:

"MAC": "00:50:56:9a:13:2a:02:1f:a0:00:0d:01:08:00",

any ideas?

also:

1- isn't the default behavior something like "get everything after value_split until field_split is found" ?

2- could we have a ignore_empty configuration option? something like

kv {
    ignore_empty => true
}
2

forgot to say that logstash-plain.log is being flooded with the error generated by the empty OUT=

[2018-07-05T12:20:26,297][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"iptables-2018.07.05", :_type=>"doc", :_routing=>nil}, #LogStash::Event:0x85585ea], :response=>{"index"=>{"_index"=>"iptables-2018.07.05", "_type"=>"doc", "_id"=>"gPsHa2QB7yvTzEaiu4if", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [OUT]", "caused_by"=>{"type"=>"illegal_state_exception", "reason"=>"Can't get text on a START_OBJECT at 1:72"}}}}}

and according to the changelog it should be correctly parsed by now?

4.1.1
bugfix: correctly handle empty values between value separator and field separator (#58)

3
4
5

Version 4.2.0 will likely ship early next week including this feature.

1 Like
6

Version 4.2.0 of the KV Filter Plugin has shipped, and will be included by default in the upcoming Logstash 6.4.0 next week.

You can also update the plugin within your existing Logstash installation:

${LOGSTASH_HOME}/bin/logstash-plugin update logstash-filter-kv
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.