XML PARSED

Diego_Gutierrez · April 15, 2019, 12:45pm

#ISSUE LOGSTASH PARSE XML

Hi, I've a problem parsing xml response from API (http_poller). I receive a weird xml response and idk how I can parse it. My file.conf is this:

The API response is(postman image):
imagen

I want to get only the six marked fields on the photo ingest in Elasticsearch. I've made this filter to parse the response:

And the response I get it is like:

Someone can help me please?

Badger · April 15, 2019, 1:04pm

Please do not post pictures of text. Just post the text.

fm.alarmObject is also an array, so you need a [0] after it.

Diego_Gutierrez · April 15, 2019, 1:21pm

Sorry, i did cause there are a lot of text.

Doesn`t work your response, maybe I've not explain good.

I want to get a final output like:

{
"@version" => "1",
"@timestamp" => 2019-04-15T13:14:05.431Z,
"severity"=>"info"
"alarmName" =>"363"
"nodeName"=> "LSTORRELODJA0"
.......
}

for each of the alarms of the response.

Thanks so much for u time

Badger · April 15, 2019, 1:50pm

You have store_xml true, so instead of using xpath you could use mutate

 mutate { add_field => { "alarmas" => "%{[Body][0][findResponse][0][Result][...and so on]}" } }

Diego_Gutierrez · April 15, 2019, 3:01pm

mmm nothing.

I get the string like "alarma" => "%{[Body][0][findResponse][0]}" but not values.

Badger · April 15, 2019, 3:23pm

So starting off with

mutate { add_field => { "alarmName" => "%{[alarmas][Body][0][findResponse][0][result][0]}" } }

what do you get. Please post text, not a picture.

Diego_Gutierrez · April 16, 2019, 8:20am

I get the following response:

logstash_testdiego | {
logstash_testdiego | "@timestamp" => 2019-04-16T08:09:39.057Z,
logstash_testdiego | "@version" => "1",
logstash_testdiego | "tags" => [
logstash_testdiego | [0] "multiline"
logstash_testdiego | ],
logstash_testdiego | "alarmaEvent" => "{"fm.AlarmObject":[{"severity":["info"],"nodeName":["LSTORRELODJA0"],"al armClassTag":["radiusaccounting.RadiusAcctPlcyFailure"],"probableCause":["260"],"alarmName":["363"],"lastTimeDetec ted":["1555352020018"],"type":["38"]},{"severity":["info"],"nodeName":["LSCANDELARGEN2"],"alarmClassTag":["n etw.FileTransferFailure"],"probableCause":["89"],

the filter is:

filter{

xml {
source => "message"
store_xml => true
target => "alarmas"
# xpath => [
# "/alarmas/Body[0]/findResponse[0]/result[0]/fm.AlarmObject[0]/alarmName", "alarm_Name"
# ]
}

mutate {

add_field => { "alarmaEvent" => "%{[alarmas][Body][0][findResponse][0][result][0]}" }
remove_field => ["message","host", "http_poller_metadata", "alarmas"]
}

}

Diego_Gutierrez · April 16, 2019, 9:37am

Now I get this:

logstash_testdiego | {
logstash_testdiego | "@version" => "1",
logstash_testdiego | "@timestamp" => 2019-04-16T09:24:37.860Z,
logstash_testdiego | "severity" => [
logstash_testdiego | [ 0] "info",
logstash_testdiego | [ 1] "info",
logstash_testdiego | [ 2] "info",
logstash_testdiego | [ 3] "info",
logstash_testdiego | [ 4] "minor",

with the next filter code:

filter{

xml {
source => "message"
remove_namespaces => true
store_xml => false
# target => "alarmas"
xpath => [
"//Header/header/requestID/text()", "IdRequest",
"//Body/findResponse/result/fm.AlarmObject/severity/text()", "severity"
]
}

if [IdRequest] { mutate { replace => { "IdRequest" => "%{[IdRequest][0]}" } } }

mutate {
remove_field => ["message","host", "http_poller_metadata"]
}

}

But i want to get a diferent doc per each severity record. Like:

logstash_testdiego | {
logstash_testdiego | "@version" => "1",
logstash_testdiego | "@timestamp" => 2019-04-16T09:24:37.860Z,
logstash_testdiego | "severity" => [
logstash_testdiego | [ 0] "info"
logstash_testdiego | }

logstash_testdiego | {
logstash_testdiego | "@version" => "1",
logstash_testdiego | "@timestamp" => 2019-04-16T09:24:37.860Z,
logstash_testdiego | "severity" => [
logstash_testdiego | [ 1] "info"
logstash_testdiego | }

....

Badger · April 16, 2019, 12:36pm

Once you have an array called severity you can us a split filter to create multiple events

split { field => "severity" }

Diego_Gutierrez · April 17, 2019, 10:46am

yes its work. But i need something more.
Because I've more one field to split.

my XML config is:

xml {
    source => "message"
    remove_namespaces => true
    store_xml => false
    target => "alarmas"
    xpath => [
            "//severity/text()", "severity",
            "//alarmName/text()", "alarmName"
  ] 

 }

So i've need diferent doc with all fields:
my input xml is:

<SOAP:Envelope xmlns:SOAP="http://schemas.xmlsoap.org/soap/envelope/">
    <SOAP:Header>
        <header xmlns="xmlapi_1.0">
            <requestID>samostats@954345</requestID>
            <requestTime>Apr 17, 2019 11:08:19 AM</requestTime>
            <responseTime>Apr 17, 2019 11:08:19 AM</responseTime>
        </header>
    </SOAP:Header>
    <SOAP:Body>
        <findResponse xmlns="xmlapi_1.0">
            <result>
                <fm.AlarmObject>
                    <severity>info</severity>
                    <probableCause>260</probableCause>
                    <alarmName>363</alarmName>
                    <type>38</type>
                    <lastTimeDetected>1555352020018</lastTimeDetected>
                    <nodeName>LSTORRELODJA0</nodeName>
                    <alarmClassTag>radiusaccounting.RadiusAcctPlcyFailure</alarmClassTag>
                    <children-Set></children-Set>
                </fm.AlarmObject>
                <fm.AlarmObject>
                    <severity>info</severity>
                    <probableCause>89</probableCause>
                    <alarmName>8152</alarmName>
                    <type>4</type>
                    <lastTimeDetected>1549357659055</lastTimeDetected>
                    <nodeName>LSCANDELARGEN2</nodeName>
                    <alarmClassTag>netw.FileTransferFailure</alarmClassTag>
                    <children-Set></children-Set>
                </fm.AlarmObject>
                <fm.AlarmObject>
                    <severity>info</severity>
                    <probableCause>400</probableCause>
                    <alarmName>528</alarmName>
                    <type>45</type>
                    <lastTimeDetected>1543426495541</lastTimeDetected>
                    <nodeName>lssamma4</nodeName>
                    <alarmClassTag>schedule.scheduledTaskCompletionStatus</alarmClassTag>
                    <children-Set></children-Set>
                </fm.AlarmObject>
            </result>
        </findResponse>
    </SOAP:Body>
</SOAP:Envelope>

I want a doc per each <fm.AlarmObject>
so i expect

{
"@version" =&gt; "1",
"@timestamp" =&gt; 2019-04-15T13:14:05.431Z,
"severity"=&gt;"info"
"alarmName" =&gt;"363"
.......
},
{
"@version" =&gt; "1",
"@timestamp" =&gt; 2019-04-15T13:14:05.431Z,
"severity"=&gt;"info"
"alarmName" =&gt;"8152"
.......
},
{
"@version" =&gt; "1",
"@timestamp" =&gt; 2019-04-15T13:14:05.431Z,
"severity"=&gt;"info"
"alarmName" =&gt;"528"
.......
}

but if I use:

split { field => "severity" }
split { field => "alarmName" }

my output is:

 {
logstash_testdiego |      "alarmName" => "363",
logstash_testdiego |     "@timestamp" => 2019-04-17T10:41:36.451Z,
logstash_testdiego |           "path" => "/usr/share/logstash/datos/gestores/file.xml",
logstash_testdiego |       "severity" => "info",
logstash_testdiego |       "@version" => "1"
logstash_testdiego | }
logstash_testdiego | {
logstash_testdiego |      "alarmName" => "8152",
logstash_testdiego |     "@timestamp" => 2019-04-17T10:41:36.451Z,
logstash_testdiego |           "path" => "/usr/share/logstash/datos/gestores/file.xml",
logstash_testdiego |       "severity" => "info",
logstash_testdiego |       "@version" => "1"
logstash_testdiego | }
logstash_testdiego | {
logstash_testdiego |      "alarmName" => "528",
logstash_testdiego |     "@timestamp" => 2019-04-17T10:41:36.451Z,
logstash_testdiego |           "path" => "/usr/share/logstash/datos/gestores/file.xml",
logstash_testdiego |       "severity" => "info",
logstash_testdiego |       "@version" => "1"
logstash_testdiego | }
logstash_testdiego | {
logstash_testdiego |      "alarmName" => "363",
logstash_testdiego |     "@timestamp" => 2019-04-17T10:41:36.451Z,
logstash_testdiego |           "path" => "/usr/share/logstash/datos/gestores/file.xml",
logstash_testdiego |       "severity" => "info",
logstash_testdiego |       "@version" => "1"
logstash_testdiego | }
logstash_testdiego | {
logstash_testdiego |      "alarmName" => "8152",
logstash_testdiego |     "@timestamp" => 2019-04-17T10:41:36.451Z,
logstash_testdiego |           "path" => "/usr/share/logstash/datos/gestores/file.xml",
logstash_testdiego |       "severity" => "info",
logstash_testdiego |       "@version" => "1"
logstash_testdiego | }
logstash_testdiego | {
logstash_testdiego |      "alarmName" => "528",
logstash_testdiego |     "@timestamp" => 2019-04-17T10:41:36.451Z,
logstash_testdiego |           "path" => "/usr/share/logstash/datos/gestores/file.xml",
logstash_testdiego |       "severity" => "info",
logstash_testdiego |       "@version" => "1"
logstash_testdiego | }
logstash_testdiego | {
logstash_testdiego |      "alarmName" => "363",
logstash_testdiego |     "@timestamp" => 2019-04-17T10:41:36.451Z,
logstash_testdiego |           "path" => "/usr/share/logstash/datos/gestores/file.xml",
logstash_testdiego |       "severity" => "info",
logstash_testdiego |       "@version" => "1"
logstash_testdiego | }
logstash_testdiego | {
logstash_testdiego |      "alarmName" => "8152",
logstash_testdiego |     "@timestamp" => 2019-04-17T10:41:36.451Z,
logstash_testdiego |           "path" => "/usr/share/logstash/datos/gestores/file.xml",
logstash_testdiego |       "severity" => "info",
logstash_testdiego |       "@version" => "1"
logstash_testdiego | }
logstash_testdiego | {
logstash_testdiego |      "alarmName" => "528",
logstash_testdiego |     "@timestamp" => 2019-04-17T10:41:36.451Z,
logstash_testdiego |           "path" => "/usr/share/logstash/datos/gestores/file.xml",
logstash_testdiego |       "severity" => "info",
logstash_testdiego |       "@version" => "1"
logstash_testdiego | }

I expect only 3 docs but i've received a combination per each field with the other. And i cant create a array with split

Badger · April 17, 2019, 12:34pm

Your post is unreadable. You XML is being interpreted as HTML tags so none of the elements actually appear in the post. Edit your posts and either select the XML (or configuration) and click on </> in the toolbar above the edit pane to treat it as a block quote, or surround the lines of XML with lines containing just three backticks...

```

Diego_Gutierrez · April 17, 2019, 12:48pm

Sorry is my first post in this discuss.
Now is better?

Badger · April 17, 2019, 2:17pm

Much better. Does

    xml {
        source => "message"
        remove_namespaces => true
        store_xml => true
        target => "alarmas"
        force_array => false
        remove_field => [ "message" ]
    }
    split { field => "[alarmas][Body][findResponse][result][fm.AlarmObject]" }

work for you?

Diego_Gutierrez · April 17, 2019, 2:21pm

I got it with the following way:

filter{
  ## interpret the message payload as XML
  xml {
    source => "message"
    remove_namespaces => true
    store_xml => false
    target => "alarmas"
    xpath => {"//fm.AlarmObject" => "alarma"}

 }

  split { field => "alarma"}

  mutate {
  gsub=>["alarma", "\n", ""]
  # split=>{"alarma" => "\n"}

  remove_field => ["message","host", "http_poller_metadata","alarmas", "tags"] 
  }

  xml {
    source => "alarma"
    remove_namespaces => true
    store_xml => false
    # target => "alarms"
    xpath => [
      "//severity/text()" , "severity",
      "//probableCause/text()" , "probableCause",
      "//alarmName/text()" , "alarmName",
      "//type/text()" , "type",
      "//alarmClassTag/text()" , "alarmClassTag",
      "//nodeName/text()" , "NodeName"
    ]

 }

 mutate{
   remove_field => ["alarma"]
 }
 mutate { replace => [ 
   "severity", "%{[severity][0]}",
   "probableCause", "%{[probableCause][0]}",
   "alarmName", "%{[alarmName][0]}",
   "type", "%{[type][0]}",
   "alarmClassTag", "%{[alarmClassTag][0]}",
   "NodeName","%{[NodeName][0]}"
   ] } 

}

maybe my way is not the best to the processing speed but i get the result like i want it:

logstash_testdiego | {
logstash_testdiego |              "type" => "38",
logstash_testdiego |     "alarmClassTag" => "radiusaccounting.RadiusAcctPlcyFailure",
logstash_testdiego |        "@timestamp" => 2019-04-17T14:11:48.664Z,
logstash_testdiego |          "@version" => "1",
logstash_testdiego |          "severity" => "info",
logstash_testdiego |     "probableCause" => "260",
logstash_testdiego |         "alarmName" => "363",
logstash_testdiego |          "NodeName" => "LSTORRELODJA0",
logstash_testdiego |              "path" => "/usr/share/logstash/datos/gestores/file.xml"
logstash_testdiego | }
logstash_testdiego | {
logstash_testdiego |              "type" => "4",
logstash_testdiego |     "alarmClassTag" => "netw.FileTransferFailure",
logstash_testdiego |        "@timestamp" => 2019-04-17T14:11:48.664Z,
logstash_testdiego |          "@version" => "1",
logstash_testdiego |          "severity" => "info",
logstash_testdiego |     "probableCause" => "89",
logstash_testdiego |         "alarmName" => "8152",
logstash_testdiego |          "NodeName" => "LSCANDELARGEN2",
logstash_testdiego |              "path" => "/usr/share/logstash/datos/gestores/file.xml"
logstash_testdiego | }
logstash_testdiego | {
logstash_testdiego |              "type" => "45",
logstash_testdiego |     "alarmClassTag" => "schedule.scheduledTaskCompletionStatus",
logstash_testdiego |        "@timestamp" => 2019-04-17T14:11:48.664Z,
logstash_testdiego |          "@version" => "1",
logstash_testdiego |          "severity" => "info",
logstash_testdiego |     "probableCause" => "400",
logstash_testdiego |         "alarmName" => "528",
logstash_testdiego |          "NodeName" => "lssamma4",
logstash_testdiego |              "path" => "/usr/share/logstash/datos/gestores/file.xml"
logstash_testdiego | }

anyway i will try with ur solution. Thanks so much to the patience with me jejejejej

system · May 15, 2019, 2:21pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Logstash xml parsing Logstash	2	623	April 5, 2017
Logstash mutate add_field Logstash	9	1968	July 6, 2017
Help Logstash XML Parsing to Xpath Logstash	23	1392	September 11, 2020
Struggling to parse XML using Logstash Logstash	1	280	October 13, 2020
How extract data with logstash filter xml from a complex xml Logstash	3	933	November 29, 2019

XML PARSED

Related topics