Filtering json fields to be outputted by logstash

[aszac (Agnieszka Szachewicz)]

Hello!

I am using logstash 5.4 to read a json interaction from kafka and then filter it so that:

1. interaction will be dropped unless field car.name exist
2. only fields car.name and car.model are outputted

My config looks like this:

input {
kafka {
topics => members
bootstrap_servers => "***"
codec => "json"
group_id => "members"
}
}
filter {
if ![car][name] {
drop { }
}
prune {
whitelist_names => [
"[car][name]",
"[car][model]"
]
}
}
output {
stdout {
codec => "json"
}
}

Input and output example:
{"name":"John","surname":"Smith","age":30,"car":{"name":"BMW","model":"320","speed":123},"job":{"company":"Tesco","title":"manager"}}
{"@timestamp":"2017-07-07T11:00:57.986Z","car":{"name":"BMW","model":"320","speed":123},"surname":"Smith","name":"John"}

How do I make the config correct, so that only the whitelisted fields are being outputted?

[magnusbaeck (Magnus Bäck)]

Unfortunately the prune filter doesn't support subfields. For now you'll have to use a ruby filter to prune the subfields of the car field.

Issue: Failure when using prune on a nested field

[aszac (Agnieszka Szachewicz)]

I have got around that problem by first using the mutate -> add_field plugin to flatten the json and then used the prune filter on the new flattened field.

e.g.
mutate {
add_field => {
"car.name" => "%{[car][name]}"
"car.model" => "%{[car][model]}"
}
}
prune {
whitelist_names => [
"car.name",
"car.model",
]
}