Global field gsub filter?

[_Ian]

Is there a way to apply a gsub action to every field during filtering?

I'm currently using dissect to split apart a message and want to do a more global replace of a null character to an actual null ES is happy with. Currently I can apply this iteratively inside a large subset of IF statemetents, but only while calling each defined field directly by name. I'd like to catch every field at once as it leaves the subset of IF dissect statements and sub out exact full field matches with a null.

I have a feeling I may end up needing ruby code to do so?

[magnusbaeck (Magnus Bäck)]

Is there a way to apply a gsub action to every field during filtering?

Not without a ruby filter.

[_Ian]

I seem to be getting caught up in this 5.0 change: https://www.elastic.co/guide/en/logstash/5.3/event-api.html

ruby { code => "event.to_hash.each { |k, v|
event[k] = v.gsub!('-', '') } "
}

Results in:

[ERROR][logstash.filters.ruby    ] Ruby exception occurred: Direct event field references (i.e. event['field'] = 'value') have been disabled in favor of using event get and set methods (e.g. event.set('field', 'value')). Please consult the Logstash 5.0 breaking changes documentation for more details.

What would be the best way to accomplish this?

[magnusbaeck (Magnus Bäck)]

Just replace

event[k] = v.gsub!('-', '')

with

event.set(k, v.gsub!('-', ''))

[_Ian]

Getting undefined method gsub! now. Is there a way to iterate through each field value with event.get?

[magnusbaeck (Magnus Bäck)]

Getting undefined method gsub! now.

Please always quote complete error message.

Is there a way to iterate through each field value with event.get?

event.get returns whatever object the field contains. gsub! will only work for string values.

[_Ian]

For anyone else looking to iterate through every field this is what we ended up using:

ruby {
   code => "
      hash = event.to_hash
      hash.each do |k,v|
        if v == nil or v == '' or v == '-' or v == ' '
          event.remove(k)
        end
   "}