Remove fields with value: nil

Elastic Stack Logstash
1

Hi,
I'm using the csv filter to split my lines up. Working quite well apart from one problem...
Some fields are being populated with a value of nil and I'd like to remove those fields:-

"SkillGroupSix" => nil

I have tried things like:

if [SkillGroupSix] =~ /nil/ {
  mutate {
      remove_field => [SkillGroupSix]
  }
}

But that doesn't work. I think it is because nil is not a string for the =~ to match on.

Would appreciate any suggestions for how to identify and match fields with a value of nil.

Thanks.

2
3

You don't mention which version of Logstash you are using, but as of LS 2.1.1 which includes logstash-filter-csv-2.1.0 there is a 'skip_empty_columns' config option.

From filter csv.rb ( ./logstash-2.1.1/vendor/bundle/jruby/1.9/gems/logstash-filter-csv-2.1.0/lib/logstash/filters/csv.rb):

Define whether empty columns should be skipped.

Defaults to false. If set to true, columns containing no value will not get set.

config :skip_empty_columns, :validate => :boolean, :default => false

This setting is listed in Logstash 2.2 reference for csv filter (but not listed in LS-2.1 reference [even though it's in LS-2.1.1)

If you are running an earlier version of Logstash (than 2.1.1), maybe updating the csv gem itself is possibility(??) :confused: -- I really don't know.

4

Thanks Magnus. That worked and got me going. The ruby filter strikes again!

5

Hi,
I'm using LS 2.2. Thanks for this ... I suspect that this would solve the problem for me but had already implemented the ruby filter fix from Magnus. I'll probably switch to this method asap though.
Many thanks!

6

It doesn't work ELK has evolved

7

It doesn't work ELK has evolved

Correct, it requires a few adjustments if you want to use it with Logstash 5.0 or later.

8

I think there isn't necessary use ruby now, logstash have many plugin ?

9

Logstash has many plugins but I don't think it has a plugin for removing field with nil values.