Remove fields with value: nil

I'm using the csv filter to split my lines up. Working quite well apart from one problem...
Some fields are being populated with a value of nil and I'd like to remove those fields:-

"SkillGroupSix" => nil

I have tried things like:

if [SkillGroupSix] =~ /nil/ {
  mutate {
      remove_field => [SkillGroupSix]

But that doesn't work. I think it is because nil is not a string for the =~ to match on.

Would appreciate any suggestions for how to identify and match fields with a value of nil.


You don't mention which version of Logstash you are using, but as of LS 2.1.1 which includes logstash-filter-csv-2.1.0 there is a 'skip_empty_columns' config option.

From filter csv.rb ( ./logstash-2.1.1/vendor/bundle/jruby/1.9/gems/logstash-filter-csv-2.1.0/lib/logstash/filters/csv.rb):

Define whether empty columns should be skipped.

Defaults to false. If set to true, columns containing no value will not get set.

config :skip_empty_columns, :validate => :boolean, :default => false

This setting is listed in Logstash 2.2 reference for csv filter (but not listed in LS-2.1 reference [even though it's in LS-2.1.1)

If you are running an earlier version of Logstash (than 2.1.1), maybe updating the csv gem itself is possibility(??) :confused: -- I really don't know.

Thanks Magnus. That worked and got me going. The ruby filter strikes again!

I'm using LS 2.2. Thanks for this ... I suspect that this would solve the problem for me but had already implemented the ruby filter fix from Magnus. I'll probably switch to this method asap though.
Many thanks!

It doesn't work ELK has evolved

It doesn't work ELK has evolved

Correct, it requires a few adjustments if you want to use it with Logstash 5.0 or later.

I think there isn't necessary use ruby now, logstash have many plugin ?

Logstash has many plugins but I don't think it has a plugin for removing field with nil values.