Good afternoon,
I am using Logstash to read in JSON documents from a Kafka topic and indexing them into Elasticsearch. Many of the fields in the document contain "null" for value. Since I don't know which fields in which documents are going to be null, I want to have a generic check that iterates through each field, dropping fields that have null values, so they don't get indexed.
Since I don't know the field names ahead of time, I don't want to write a check that explicitly uses a field name, but iterates through all the fields in the doc. So not something like if [user_id] == null, but something like if[field_N] == null then drop field_N, where N = 1 through number of fields in the doc.
Initially thought prune might be the answer but doesn't look like that's a good fit for this.
Any help is appreciated.
Thanks.