Iterating through all fields in a JSON message, checking for null values

[truesecure (John Jack)]

Good afternoon,

I am using Logstash to read in JSON documents from a Kafka topic and indexing them into Elasticsearch. Many of the fields in the document contain "null" for value. Since I don't know which fields in which documents are going to be null, I want to have a generic check that iterates through each field, dropping fields that have null values, so they don't get indexed.

Since I don't know the field names ahead of time, I don't want to write a check that explicitly uses a field name, but iterates through all the fields in the doc. So not something like if [user_id] == null, but something like if[field_N] == null then drop field_N, where N = 1 through number of fields in the doc.

Initially thought prune might be the answer but doesn't look like that's a good fit for this.

Any help is appreciated.

Thanks.

[magnusbaeck (Magnus Bäck)]

Unfortunately the prune filter can't remove null values (but it probably should) so for now you'll have to use a ruby filter.