Parsing only a limited set of fields in JSON

Elastic Stack Logstash
1

I need to parse a JSON log message but keep only a few fields. Is it possible to couple the JSON filter plugin (https://www.elastic.co/guide/en/logstash/current/plugins-filters-json.html) with the KV filter plugin (https://www.elastic.co/guide/en/logstash/current/plugins-filters-kv.html) to do so?

My idea was to use the include_keys command from KV.

2

No, you can't do that. Perhaps the prune filter can be useful.

1 Like
3

That's what I feared. Here's my (non-working) attempt to keep only the JSON keys "foo" and "bar":

filter {
                json {
                        source => "message"
                        target => "mylog"
                        remove_field => [ "message", "source", "fields", "[mylog][@version]", "[mylog][type]" ]
                }

                kv {
                        source => "message"
                        field_split => ","
                        value_split => ":"
                        include_keys => [ "mylog.@fields.foo", "mylog.@fields.bar" ]
                }
}

This filter parses correctly the JSON but includes all values, which isn't what I want.
I'll try the prune filter. Thanks for now.

4

Prune should work, but you could also copy the fields out of [mylog] then delete it.

mutate { copy => { "[mylog][foo]" => "foo" "[mylog][bar]" => "bar" } }
mutate { remove_field => [ "[mylog]" ] }
mutate { rename => { "foo" => "[mylog][foo]" "bar" => "[mylog][bar]" } }
1 Like
5

Thank you all. I managed to do what I wanted. Here's the final code:

filter {
                json {
                        source => "message"
                        target => "mylog"
                        remove_field => [ "message", "source", "fields", "[mylog][@version]", "[mylog][type]" ]
                }

                kv {
                        source => "message"
                        field_split => ","
                        value_split => ":"
                }

                date {
                        match => [ "[mylog][@timestamp]", "ISO8601" ]
                        remove_field => [ "[mylog][@timestamp]" ]
                }

                mutate {
                        copy => { "[mylog][@fields][type]" => "mylog_type" }
                        copy => { "[mylog][@fields][action]" => "mylog_action" }
                        remove_field => [ "[mylog]" ]
                }
}

There's probably redundancy between the json and the kv code so some parts could be ditched out. Also, remove_field => [ "[mylog][@timestamp]" ] is unnecessary as the filter removes the whole [mylog] field further on. Feel free to share your thoughts.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.