XML Filter :How to create array of objects using xml filter (xpath)

Elastic Stack Logstash
1

I am new to logstash. I am trying push XML (generated as the output of http polling) in logstash. I want to have the XML elements attributes convert to json objects.I am using xpath for parsing xml data.But i am getting separate array of xpath fields insteda of array of objects.

My http response will be like this:

17 AGFAM Falmouth Harbor P CARB AG -4.0000 1 18 GPIDS Iles Des Saintes P CARB GP 0.0000 1 19 MQQMR Le Marin P CARB MQ -4.0000 1

My logstach confi xml filter like this:

filter {

xml {
store_xml => "false"
source => "message"
target => "abc"
remove_field => ["message","http_poller_metadata","@version","@timestamp"]
xpath => [
"/ResListLocation/LocationList/Location/LocationId/text()", "id",
"/ResListLocation/LocationList/Location/Name/text()", "name",
"/ResListLocation/LocationList/Location/Code/text()", "code"

]
force_array => false

}
}

I am gertting output like this:

 {
                "_index": "logst41",
                "_type": "port",
                "_id": "AWUlK6ZzTgnFfJZNCIK6",
                "_score": 1,
                "_source": {
                    "code": [
                        "AGFAM",
                        "GRKAP",
                        "SIKOP",
                        "MEKOT"
                    ],
                    "name": [
                        "Salem, Massachusetts",
                        "Bridgetown",
                        "Santa Marta",
                        "Tampa"
                    ],
                    "id": [
                        "17",
                        "18",
                        "19",
                        "20"
                    ]
                }
}

But I am expecting output like this:

{
"_index": "logst41",
"_type": "port",
"_id": "AWUlK6ZzTgnFfJZNCIK6",
"_score": 1,
"_source": {
abc:[
{"code":"AGFAM", "name":"Salem, Massachusetts","id": "17"},
{"code":"GRKAP", "name":"Bridgetown","id": "18"},
{"code":"SIKOP", "name":"Santa Marta","id": "19"}
]
}

}
2 
<ResListLocation>
    <Result>
        <Status>A</Status>
    </Result>
    <LocationList>
        <Location>
            <LocationId>17</LocationId>
            <Code>AGFAM</Code>
            <Name>Falmouth Harbor</Name>
            <Type>P</Type>
            <Region>CARB</Region>
            <Country>AG</Country>
            <TimeZone>-4.0000</TimeZone>
            <Enabled>1</Enabled>
        </Location>
        <Location>
            <LocationId>18</LocationId>
            <Code>GPIDS</Code>
            <Name>Iles Des Saintes</Name>
            <Type>P</Type>
            <Region>CARB</Region>
            <Country>GP</Country>
            <TimeZone>0.0000</TimeZone>
            <Enabled>1</Enabled>
        </Location>
        <Location>
            <LocationId>19</LocationId>
            <Code>MQQMR</Code>
            <Name>Le Marin</Name>
            <Type>P</Type>
            <Region>CARB</Region>
            <Country>MQ</Country>
            <TimeZone>-4.0000</TimeZone>
            <Enabled>1</Enabled>
        </Location>
</LocationList>
</ResListLocation>
3

Yeah... that's how the filter works. You can use a ruby filter to join the elements of each array into an array of objects. I believe examples of this have been posted in the past.

4

Something like this:

    ruby { code => "
        c = event.get('code')
        i = event.get('id')
        n = event.get('name')
        a = []
        c.each_index { |k|
            h = { 'code' => c[k], 'id' => i[k], 'name' => n[k] }
            a << h
        }
        event.set('arrayOfHashes', a)
    "

Error handling is left as an exercise for the reader.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.