Event.remove not working

Elastic Stack Logstash
1

I am using the following ruby filter to update the fields.

def filter(event)
  keys = event.to_hash.keys
  keys.each{|key|

    if ( key.start_with? '_' and key != "_@timestamp")
      value = event.get(key)
      newkey = key.sub!(/^_/, '')
      event.remove(key)
      event.set(newkey , value)

    elsif (key == "_@timestamp")
      value = event.get(key)
      event.set("@apptimestamp", Time.at(value))
      event.remove(key)
    end
      
  }
  return [event]

  rescue Exception => e
    event.set('logstash_ruby_exception', 'underscores: ' + e.message)
    return [event]
end

The code duplicates fields starting with underscore, to fields not starting with underscore. But for some reason doesn't eliminate the one's starting with underscore. That is, event.remove doesn't work as expected. Any ideas?

Thanks in advance!

2

What version of Logstash?

3

6.2.4
image used is docker.elastic.co/logstash/logstash:6.2.4

4

sub! alters the value of key so it is not removable.
In Ruby IRB...

irb(main):001:0> key = "_abc"
=> "_abc"
irb(main):002:0> key.sub!(/^_/, "")
=> "abc"
irb(main):003:0> key
=> "abc"

Looking at how the mutate filter works and with my knowledge of ruby I suggest this improvement.

input {
  generator {
    message => '{"_a": "A", "_b": "B", "c_": "C"}'
    count => 1
  }
}

filter {
  json {
    source => "message"
  }
  if "_jsonparsefailure" not in [tags]  {
    ruby {
      code => '
        keys = event.to_hash.keys
        keys.select{|key| key.start_with?("_") && key != "_@timestamp" }.each do |key|
          newkey = key[1, key.length] # we know the first character is an underscore
          event.set(newkey, event.remove(key))
        end
        # deal with _@timestamp here, not inside the iterator
      '
    }
  }
}

output {
  stdout {
    codec => rubydebug
  }
}

Gives

{
      "sequence" => 0,
             "a" => "A",
             "b" => "B",
    "@timestamp" => 2018-06-01T09:57:42.934Z,
      "@version" => "1",
          "host" => "Elastics-MacBook-Pro.local",
       "message" => "{\"_a\": \"A\", \"_b\": \"B\", \"c_\": \"C\"}",
            "c_" => "C"
}
5

Thank you so much! I am so silly, didn't notice that! :smiley:

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.