# Iterating through all fields in a JSON message, checking for null values

**URL:** <https://discuss.elastic.co/t/iterating-through-all-fields-in-a-json-message-checking-for-null-values/95776>\
**Category:** Logstash\
**Created:** [August 3, 2017, 8:36pm UTC](https://discuss.elastic.co/t/iterating-through-all-fields-in-a-json-message-checking-for-null-values/95776 "2017-08-03T20:36:55Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![truesecure](https://avatars.discourse-cdn.com/v4/letter/t/f07891/32.png) [@truesecure](https://discuss.elastic.co/u/truesecure)\
**Post date:** [August 3, 2017, 8:36pm UTC](https://discuss.elastic.co/t/iterating-through-all-fields-in-a-json-message-checking-for-null-values/95776/1 "2017-08-03T20:36:56Z")

</div>

Good afternoon,

I am using Logstash to read in JSON documents from a Kafka topic and indexing them into Elasticsearch. Many of the fields in the document contain "null" for value. Since I don't know which fields in which documents are going to be null, I want to have a generic check that iterates through each field, dropping fields that have null values, so they don't get indexed.

Since I don't know the field names ahead of time, I don't want to write a check that explicitly uses a field name, but iterates through all the fields in the doc. So not something like if [user\_id] == null, but something like if[field\_N] == null then drop field\_N, where N = 1 through number of fields in the doc.

Initially thought prune might be the answer but doesn't look like that's a good fit for this.

Any help is appreciated.

Thanks.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 4, 2017, 6:06am UTC](https://discuss.elastic.co/t/iterating-through-all-fields-in-a-json-message-checking-for-null-values/95776/2 "2017-08-04T06:06:53Z")

</div>

Unfortunately the prune filter can't remove null values (but it probably should) so for now you'll have to use a ruby filter.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 1, 2017, 6:07am UTC](https://discuss.elastic.co/t/iterating-through-all-fields-in-a-json-message-checking-for-null-values/95776/3 "2017-09-01T06:07:06Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.