Event.remove not working


(Akash Kumar Dutta) #1

I am using the following ruby filter to update the fields.

def filter(event)
  keys = event.to_hash.keys
  keys.each{|key|

    if ( key.start_with? '_' and key != "_@timestamp")
      value = event.get(key)
      newkey = key.sub!(/^_/, '')
      event.remove(key)
      event.set(newkey , value)

    elsif (key == "_@timestamp")
      value = event.get(key)
      event.set("@apptimestamp", Time.at(value))
      event.remove(key)
    end
      
  }
  return [event]

  rescue Exception => e
    event.set('logstash_ruby_exception', 'underscores: ' + e.message)
    return [event]
end

The code duplicates fields starting with underscore, to fields not starting with underscore. But for some reason doesn't eliminate the one's starting with underscore. That is, event.remove doesn't work as expected. Any ideas?

Thanks in advance!


(Guy Boertje) #2

What version of Logstash?


(Akash Kumar Dutta) #3

6.2.4
image used is docker.elastic.co/logstash/logstash:6.2.4


(Guy Boertje) #4

sub! alters the value of key so it is not removable.
In Ruby IRB...

irb(main):001:0> key = "_abc"
=> "_abc"
irb(main):002:0> key.sub!(/^_/, "")
=> "abc"
irb(main):003:0> key
=> "abc"

Looking at how the mutate filter works and with my knowledge of ruby I suggest this improvement.

input {
  generator {
    message => '{"_a": "A", "_b": "B", "c_": "C"}'
    count => 1
  }
}

filter {
  json {
    source => "message"
  }
  if "_jsonparsefailure" not in [tags]  {
    ruby {
      code => '
        keys = event.to_hash.keys
        keys.select{|key| key.start_with?("_") && key != "_@timestamp" }.each do |key|
          newkey = key[1, key.length] # we know the first character is an underscore
          event.set(newkey, event.remove(key))
        end
        # deal with _@timestamp here, not inside the iterator
      '
    }
  }
}

output {
  stdout {
    codec => rubydebug
  }
}

Gives

{
      "sequence" => 0,
             "a" => "A",
             "b" => "B",
    "@timestamp" => 2018-06-01T09:57:42.934Z,
      "@version" => "1",
          "host" => "Elastics-MacBook-Pro.local",
       "message" => "{\"_a\": \"A\", \"_b\": \"B\", \"c_\": \"C\"}",
            "c_" => "C"
}

(Akash Kumar Dutta) #5

Thank you so much! I am so silly, didn't notice that! :smiley:


(system) #6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.