# Remove fields with value: nil

**URL:** <https://discuss.elastic.co/t/remove-fields-with-value-nil/42243>\
**Category:** Logstash\
**Created:** [February 19, 2016, 1:22pm UTC](https://discuss.elastic.co/t/remove-fields-with-value-nil/42243 "2016-02-19T13:22:53Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kryten](https://avatars.discourse-cdn.com/v4/letter/k/58956e/32.png) [@Kryten](https://discuss.elastic.co/u/Kryten)\
**Post date:** [February 19, 2016, 1:22pm UTC](https://discuss.elastic.co/t/remove-fields-with-value-nil/42243/1 "2016-02-19T13:22:53Z")

</div>

Hi,  
I'm using the csv filter to split my lines up. Working quite well apart from one problem...  
Some fields are being populated with a value of nil and I'd like to remove those fields:-

`"SkillGroupSix" => nil`

I have tried things like:

```
if [SkillGroupSix] =~ /nil/ {
  mutate {
      remove_field => [SkillGroupSix]
  }
}

```

But that doesn't work. I think it is because nil is not a string for the =~ to match on.

Would appreciate any suggestions for how to identify and match fields with a value of nil.

Thanks.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 19, 2016, 1:55pm UTC](https://discuss.elastic.co/t/remove-fields-with-value-nil/42243/2 "2016-02-19T13:55:35Z")

</div>

> <https://stackoverflow.com/questions/28957870/how-to-remove-all-fields-with-null-value-in-logstash-filter>

---

<div class="post-metadata">

**Author:** ![fh8510](https://avatars.discourse-cdn.com/v4/letter/f/b487fb/32.png) [@fh8510](https://discuss.elastic.co/u/fh8510)\
**Post date:** [February 19, 2016, 5:26pm UTC](https://discuss.elastic.co/t/remove-fields-with-value-nil/42243/3 "2016-02-19T17:26:06Z")

</div>

You don't mention which version of Logstash you are using, but as of LS 2.1.1 which includes logstash-filter-csv-2.1.0 there is a 'skip\_empty\_columns' config option.

From filter csv.rb ( ./logstash-2.1.1/vendor/bundle/jruby/1.9/gems/logstash-filter-csv-2.1.0/lib/logstash/filters/csv.rb):

> # Define whether empty columns should be skipped.
> 
> # Defaults to false. If set to true, columns containing no value will not get set.
> 
> config :skip\_empty\_columns, :validate =\> :boolean, :default =\> false

This setting is listed in [Logstash 2.2 reference for csv filter](https://www.elastic.co/guide/en/logstash/2.2/plugins-filters-csv.html) (but not listed in LS-2.1 reference [even though it's in LS-2.1.1)

If you are running an earlier version of Logstash (than 2.1.1), maybe updating the[csv gem](https://rubygems.org/gems/logstash-filter-csv/versions) itself is possibility(??) 😕 -- I really don't know.

---

<div class="post-metadata">

**Author:** ![Kryten](https://avatars.discourse-cdn.com/v4/letter/k/58956e/32.png) [@Kryten](https://discuss.elastic.co/u/Kryten)\
**Post date:** [February 19, 2016, 9:44pm UTC](https://discuss.elastic.co/t/remove-fields-with-value-nil/42243/4 "2016-02-19T21:44:44Z")

</div>

Thanks Magnus. That worked and got me going. The ruby filter strikes again!

---

<div class="post-metadata">

**Author:** ![Kryten](https://avatars.discourse-cdn.com/v4/letter/k/58956e/32.png) [@Kryten](https://discuss.elastic.co/u/Kryten)\
**Post date:** [February 19, 2016, 9:46pm UTC](https://discuss.elastic.co/t/remove-fields-with-value-nil/42243/5 "2016-02-19T21:46:32Z")

</div>

Hi,  
I'm using LS 2.2. Thanks for this ... I suspect that this would solve the problem for me but had already implemented the ruby filter fix from Magnus. I'll probably switch to this method asap though.  
Many thanks!

---

<div class="post-metadata">

**Author:** ![Beuhlet\_Reseau](https://avatars.discourse-cdn.com/v4/letter/b/e95f7d/32.png) [@Beuhlet\_Reseau](https://discuss.elastic.co/u/Beuhlet_Reseau)\
**Post date:** [March 7, 2017, 10:22am UTC](https://discuss.elastic.co/t/remove-fields-with-value-nil/42243/6 "2017-03-07T10:22:14Z")

</div>

It doesn't work ELK has evolved

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 7, 2017, 10:28am UTC](https://discuss.elastic.co/t/remove-fields-with-value-nil/42243/7 "2017-03-07T10:28:23Z")

</div>

> It doesn't work ELK has evolved

Correct, it requires a few adjustments if you want to use it with Logstash 5.0 or later.

---

<div class="post-metadata">

**Author:** ![Beuhlet\_Reseau](https://avatars.discourse-cdn.com/v4/letter/b/e95f7d/32.png) [@Beuhlet\_Reseau](https://discuss.elastic.co/u/Beuhlet_Reseau)\
**Post date:** [March 7, 2017, 10:32am UTC](https://discuss.elastic.co/t/remove-fields-with-value-nil/42243/8 "2017-03-07T10:32:00Z")

</div>

I think there isn't necessary use ruby now, logstash have many plugin ?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 7, 2017, 10:51am UTC](https://discuss.elastic.co/t/remove-fields-with-value-nil/42243/9 "2017-03-07T10:51:09Z")

</div>

Logstash has many plugins but I don't think it has a plugin for removing field with nil values.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:28am UTC](https://discuss.elastic.co/t/remove-fields-with-value-nil/42243/10 "2017-07-06T04:28:05Z")

</div>