XML Filter Output to Ruby Code

[chayelastic]

Hello,

In my Logstash config, I have an XML filter with xpath that allows me to read the XML tags and store in a variable/field. It works well.
Next step, I had to use a Ruby Filter for I need to transpose a couple of arrays from the XML structure.

My only problem is How do I check for Null XML elements ie. XMLs elements are not present in the client request, before calling transpose in the below ruby code as in call SET for subscriberArray for only those Lists which are Not NULL.(Nil)

FYI. I could have upto 10 elements that I need to check for before transposing them.

`
ruby

        {

            code => "event.set('subscriberArray', [ event.get('requestList1'), event.get('requestList2') , event.get('requestList3') ].transpose)

        }

`
Thank you in advance.

[Badger]

If I understand the ask correctly you could do something like

code => '
    a = []
    [ "requestList1", "requestList2", "requestList3" ].each { |k|
        v = event.get(k)
        if v
            a << v
        end
    }
    if a != []
        event.set("subscriberArray", a.transpose)
    end
'

[chayelastic]

@Badger

code => '
    a = []
    [ "requestList1", "requestList2", "requestList3" ].each { |k|
        v = event.get(k)
        if v
            a << v
        end
    }
    if a != []
        event.set("subscriberArray", a.transpose)
    end
'

Hi Badger,

I Tested with couple of elements and it WORKS!! This is fabulous.. let me try with the real XMLs and see where I land. Clearly this shows the lack of my Ruby Skills- and I wasn't able to get past tidentifier, end of file, syntax etc...errors..

Thank you so much - Made my Friday More Fun!!

[Badger]

@chayelastic

I wasn't able to get past tidentifier, end of file, syntax etc...errors.

The error messages from the ruby compiler are not very helpful. And it does not help that most ruby errors result in logstash exiting, so each error is a minute of your life that you will not get back whilst logstash restarts

[chayelastic]

.. totally..must have done 100's of times.. and I have just started ...hopefully the journey gets better..

[chayelastic]

Hi again,

So I tried with some real XML, the problem here is the Request List3 has 4 elements where was request 1 and 2 have 2 elements only and I get a Ruby exception.

The List1 and List 2 extracts from first XML block and second XML block and stores the XML element data- I get one element from each block.
The List 3 extracts the data from first and second XML block and gets 2 elements each.

my final array should have element 1 from List 1, element 2 from List 2 and first elements from List 3 after the transpose.

Is that possible? or I have to parse the List 3 and combine the paired data before invoking the loop you suggested?

Eg. List 1. : 1a and 1b
List 2 : 2a and 2b
List 3 : 3a1, 3a2 and 3b1 and 3b2

My final expectation is [1a,2a, {3a1, 3a2}] and [1b,2b, {3b1, 3b2}].

Thank you in advance.

[Badger]

I think you will have to parse list3 to get it into the form you want.

[chayelastic]

But only if I know where to stop when I parse them.

SO XML Block 1 has say account 1, Subscription code, and underneath multiple subscribers child nodes.
I am currently using xpath to get all the accounts, subscription codes and subscribers in their lists. and then during transpose.. I get the error as Subscribers are 1 to many and Subscribers List is not the same size as account and subscription code.

Looking for ideas..