Sometimes "Ruby exception occurred: no implicit conversion of nil into String"

[vb4t]

Hello,

I use following ruby code in ruby filter to remove empty fields and fields containing "-" from fields nested in "parentfield" . The code works 99% time well, but sometimes I get ruby exception "Ruby exception occurred: no implicit conversion of nil into String". What is wrong with this code?

  ruby {
    code => "
      def walk_hash(parent, path, hash)
        path << parent if parent
        hash.each do |key, value|
          walk_hash(key, path, value) if value.is_a?(Hash)
          @paths << (path + [key]).map {|p| '[' + p + ']' }.join('')
        end
        path.pop
      end

      @paths = []
      eventhash = event.to_hash
      if eventhash.has_key?('parentfield') && eventhash['parentfield'].is_a?(Hash)
        walk_hash('parentfield', [], eventhash['parentfield'])
      end

      @paths.each do |path|
        value = event.get(path)
        event.remove(path) if value.nil? || (value.respond_to?(:empty?) && value.empty?) || (value.is_a?(String) && value == '-')
      end
    "
  }

[Badger]

Can you show us a document that triggered that ruby exception?

[vb4t]

...

It seems if I change this line:
event.remove(path) if value.nil? || (value.respond_to?(:empty?) && value.empty?) || (value.is_a?(String) && value == '-')
to this:
event.remove(path) if value.nil? || (value.respond_to?(:empty?) && value.empty?)
the issue disappears. But this will not do what I want. It will just remove empty fields (yes, almost all '-' fields shown in kibana means the value is null, but sometimes there is a - char).

[Badger]

@vb4t

(value.is_a?(String) && value == '-')

So have a second event.remove(path) that is conditional on the value not being nil and also meeting that condition.

[vb4t]

Yes, I will try. But should not be event.remove(path) if value.nil? || (value.respond_to?(:empty?) && value.empty?) || (value.is_a?(String) && value == '-') working?

[Badger]

You mean that if value is nil you would expect if value.nil? to evaluate to true, resulting in all the ||'d conditions being short-circuited so that && value == '-' never gets evaluated? You might expect that, but you are telling me it does not work that way

[vb4t]

Even if || will be processed all (which should not), (value.is_a?(String) && value == '-') contains condition to ensure value is String.
But I tried to do it in separate condition, so:

  unless value.nil?
    event.remove(path) if (value.is_a?(String) && value == '-')
  end

It causes the same issue.

[Badger]

If I wrap that object in { "parentfield" : ... } then with 6.2.4 your ruby script strips out all the nulls without throwing an exception.

[vb4t]

The content of parentfield is not the raw json, but json decoded data, so there is parentfield.applicaitonname, parentfield.operation_o.id, ...

[vb4t]

It can take about an hour and lot of events before I can see ruby exception. So it does not occur all the time.

[Badger]

If @paths is shared between threads it could get stepped on. Have you tried --pipeline.workers 1 ?

[vb4t]

It can not be shared between threads. @var is instance variable. I hope instance var is per thread (or is it per plugin instance globally?). There are too many events to be handled by just 1 worker. Also it could lead to a much longer time for the issue to occur so I can not test this.
I will try to change @paths to local variable (paths).

[vb4t]

Hmm, it looks like there is one instance of one plugin configuration for each pipeline across all pipeline threads so probably @instance vars are shared ...

[Badger]

@vb4t

so probably @instance vars are shared

Yes, they are. You can demonstrate that using this (after setting pipeline.batch.size: 1)

input { generator { count => 3 message => '' } }

filter {
    ruby {
        code => "
            if @a.nil?
                @a = rand
            end
            event.set('a', @a)
            event.set('id', Thread.current.object_id)
        "
    }
}
output { stdout { codec => rubydebug } }

The first two events may have different random numbers because they both test @a is nil at the same time, but after that every event has the same random number. For example...

{
             "a" => 0.788757598293314,
      "sequence" => 0,
    "@timestamp" => 2018-06-11T18:44:59.526Z,
            "id" => 2012
}
{
             "a" => 0.7908474711636193,
      "sequence" => 1,
    "@timestamp" => 2018-06-11T18:44:59.529Z,
            "id" => 2010
}
{
             "a" => 0.788757598293314,
      "sequence" => 3,
    "@timestamp" => 2018-06-11T18:44:59.529Z,
            "id" => 2010
}
{
             "a" => 0.788757598293314,
      "sequence" => 2,
    "@timestamp" => 2018-06-11T18:44:59.529Z,
            "id" => 2012
}

[vb4t]

I can confirm after change to local vars it is working correctly. Thanks very much. This is the working solution:

  ruby {
    code => "
      def walk_hash(parent, path, hash, paths)
        path << parent if parent
        hash.each do |key, value|
          walk_hash(key, path, value, paths) if value.is_a?(Hash)
          paths << (path + [key]).map {|p| '[' + p + ']' }.join('')
        end
        path.pop
      end

      eventhash = event.to_hash
      if eventhash.has_key?('SOMEFIELD') && eventhash['SOMEFIELD'].is_a?(Hash)
        paths = []
        walk_hash('SOMEFIELD', [], eventhash['SOMEFIELD'], paths)
        paths.each do |path|
          value = event.get(path)
          event.remove(path) if value.nil? || (value.respond_to?(:empty?) && value.empty?) [+ '-' condition if wanted]
        end
      end
    "
  }