XML filter - processing of an array of values

Elastic Stack Logstash
1

We're processing some xml events using the xml filter.
The filter setup it's quite simple:

filter {
    xml {
      source => "message"
      target => "xml"
    }
}

This filter generates two fields with an array of values:
xml.results.Name: ["OrderId","CaseId","Timestamp"]
xml.results.Value: ["123456","987654","1629106126536"]

The goal is to create new fields based on the Name and Value:
xml.results.Name.OrderId: "123456"
xml.results.Name.CaseId: "987654"
...

As there are hundreds of different field names the "xpath" definition is not good solution for us.
Is there an option in logstash how to do it?

So far we tried to do it with a ruby script:

  ruby {
        code => '
          v = event.get("[xml][results][Value]")
          k = event.get("[xml][results][Name]")
          k.each_index { |i|
            event.set("[xml][results][Name][" + k[i] + "]", v[i] ) 
          }
        '
      }

But it did not worked because of following error:

Ruby exception occurred: undefined method `each_with_index' for nil:NilClass

Thanks! :slight_smile:

2

That is telling you that k is nil, which means you are processing an event where the [xml][results][Name] field does not exist. I would wrap the k.each_index with

if k.method_defined?(each_index) and v.method_defined?(each_index) {
...
}
3

Thanks for the hint!

I tried this:

  ruby {
        code => '
          v = event.get("[xml][results][Value]")
          k = event.get("[xml][results][Name]")
		  if k.method_defined?(each_index) and v.method_defined?(each_index) {
            k.each_index { |i|
                event.set("[xml][results][Name][" + k[i] + "]", v[i] ) 
            }
		  }
        '
      }

But following error appeared:

[2021-08-17T11:29:13,485][FATAL][org.logstash.Logstash    ] Logstash stopped processing because of an error: (SyntaxError) (ruby filter code):11: syntax error, unexpected end-of-file

org.jruby.exceptions.SyntaxError: (SyntaxError) (ruby filter code):11: syntax error, unexpected end-of-file

        at org.jruby.RubyKernel.eval(org/jruby/RubyKernel.java:1048) ~[jruby-complete-9.2.13.0.jar:?]

The second question is why the field does not exist? We can see the field name and an array of values in the event json (as described in the 1st post).

4

My bad. A Ruby if statement does not take a code block. It should be

if k.method_defined?(each_index) and v.method_defined?(each_index)
...
end

Since [xml][results][Name] already exists as an array the event.set may fail telling you that it cannot coerce a string to an integer. You may need to

k = event.get("[xml][results][Name]")
event.remove("[xml][results][Name]")

Personally I would use string interpolation, although your way should also work

event.set("[xml][results][Name][#{k[i]}]", v[i] )
5

Thanks a lot! This did the trick at the end.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.