Ruby Filter to clean nulls and nils

Elastic Stack Logstash
1

I grabbed a function from a thread somewhere that uses ruby to "clean" off nulls, nills, dashes, etc so that I dont needlessly clutter ES or import them. It has worked well up until now. Now however, I have a bit of a nasty data string to deal with and it is getting hung up on a square bracket I am certain. I would prefer not to have to alter my dataset at the source, but rather adjust ruby to handle the case. Can anyone give me any help here(total novice at ruby)?

my ruby script:

    def register(params)
end

def removeEmptyField(event,h,name)
  h.each do |k,v|
   if (v.is_a?(Hash) || v.is_a?(Array)) && v.to_s != '{}'
      removeEmptyField(event,v,String.new(name.to_s) << '[' << k.to_s << ']')
    else
    if v == '' || v.to_s == '-' || v == nil || v.to_s == '{}'
      event.remove(String.new(name.to_s) << '[' << k.to_s << ']')
    end
   end
  end
end

def filter(event)
  removeEmptyField event,event.to_hash,''
  return [event]
end

Sample from my dataset:

 "file": [
        {
          "hash": {
            "md5": "2df1c30b185cd27a5f9178cac90fd4be",
            "sha256": "c33371f6757b0e726f13275185a4d79ff76b04230340574a41c0fec115822e35"
          },
          "name": "text.txt"
        },
        {
          "hash": {
            "md5": "58b1ed6c85d1930b44a4344c081bd6d0",
            "sha256": "e403f623f1411ee6174a3d7405eeeefa39a77c87477a9de0c036b0557c6c1c74"
          },
          "name": "20200908報告.zip"
        },
        {
          "name": "20200908/yCLXz\\.xls-436626853"
        },
        {
          "name": "20200908/yVzH29P[X \\.xls-3644256286"
        },
        {
          "name": "20200908/yVzyNBXz\\.xls-275462381"
        }
      ]

How I call it, just for good measure:

ruby {
        path => "/etc/logstash/ruby/cleanFields.rb"
        tag_on_exception => [ "_rubyfailure", "_ruby_cleanFields" ]
      }

And my error... which is a brutal process restarter...

 An unexpected error occurred! {:error=>#<TypeError: **no implicit conversion of Array into String**>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-ruby-3.1.5/lib/logstash/filters/ruby.rb:103:in `file_script'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-filter-ruby-3.1.5/lib/logstash/filters/ruby.rb:84:in `filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:143:in `do_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:162:in `block in multi_filter'", "org/jruby/RubyArray.java:1800:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:159:in `multi_filter'", "org/logstash/config/ir/compiler/AbstractFilterDelegatorExt.java:115:in `multi_filter'", "(eval):69132:in `block in initialize'", "org/jruby/RubyArray.java:1800:in `each'", "(eval):69122:in `block in initialize'", "(eval):69146:in `block in initialize'", "org/jruby/RubyArray.java:1800:in `each'", "(eval):69143:in `block in initialize'", "(eval):10790:in `block in filter_func'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:362:in `filter_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:341:in `worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:308:in `block in start_workers'"]}

So, I am aware that the problem is how it is handling an array, but I do not understand how I would fix this error. Help is appreciated.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.