XML PARSED


(Diego Gutierrez) #1

#ISSUE LOGSTASH PARSE XML

Hi, I've a problem parsing xml response from API (http_poller). I receive a weird xml response and idk how I can parse it. My file.conf is this:

The API response is(postman image):
imagen

I want to get only the six marked fields on the photo ingest in Elasticsearch. I've made this filter to parse the response:

And the response I get it is like:

Someone can help me please? :slight_smile:


#2

Please do not post pictures of text. Just post the text.

fm.alarmObject is also an array, so you need a [0] after it.


(Diego Gutierrez) #3

Sorry, i did cause there are a lot of text.

Doesn`t work your response, maybe I've not explain good.

I want to get a final output like:

{
"@version" => "1",
"@timestamp" => 2019-04-15T13:14:05.431Z,
"severity"=>"info"
"alarmName" =>"363"
"nodeName"=> "LSTORRELODJA0"
.......
}

for each of the alarms of the response.

Thanks so much for u time :slight_smile:


#4

You have store_xml true, so instead of using xpath you could use mutate

 mutate { add_field => { "alarmas" => "%{[Body][0][findResponse][0][Result][...and so on]}" } }

(Diego Gutierrez) #5

mmm nothing.

I get the string like "alarma" => "%{[Body][0][findResponse][0]}" but not values.


#6

So starting off with

mutate { add_field => { "alarmName" => "%{[alarmas][Body][0][findResponse][0][result][0]}" } }

what do you get. Please post text, not a picture.


(Diego Gutierrez) #7

I get the following response:

logstash_testdiego | {
logstash_testdiego | "@timestamp" => 2019-04-16T08:09:39.057Z,
logstash_testdiego | "@version" => "1",
logstash_testdiego | "tags" => [
logstash_testdiego | [0] "multiline"
logstash_testdiego | ],
logstash_testdiego | "alarmaEvent" => "{"fm.AlarmObject":[{"severity":["info"],"nodeName":["LSTORRELODJA0"],"al armClassTag":["radiusaccounting.RadiusAcctPlcyFailure"],"probableCause":["260"],"alarmName":["363"],"lastTimeDetec ted":["1555352020018"],"type":["38"]},{"severity":["info"],"nodeName":["LSCANDELARGEN2"],"alarmClassTag":["n etw.FileTransferFailure"],"probableCause":["89"],

the filter is:

filter{

xml {
source => "message"
store_xml => true
target => "alarmas"
# xpath => [
# "/alarmas/Body[0]/findResponse[0]/result[0]/fm.AlarmObject[0]/alarmName", "alarm_Name"
# ]
}

mutate {

add_field => { "alarmaEvent" => "%{[alarmas][Body][0][findResponse][0][result][0]}" }
remove_field => ["message","host", "http_poller_metadata", "alarmas"]
}

}


(Diego Gutierrez) #8

Now I get this:

logstash_testdiego | {
logstash_testdiego | "@version" => "1",
logstash_testdiego | "@timestamp" => 2019-04-16T09:24:37.860Z,
logstash_testdiego | "severity" => [
logstash_testdiego | [ 0] "info",
logstash_testdiego | [ 1] "info",
logstash_testdiego | [ 2] "info",
logstash_testdiego | [ 3] "info",
logstash_testdiego | [ 4] "minor",

with the next filter code:

filter{

xml {
source => "message"
remove_namespaces => true
store_xml => false
# target => "alarmas"
xpath => [
"//Header/header/requestID/text()", "IdRequest",
"//Body/findResponse/result/fm.AlarmObject/severity/text()", "severity"
]
}

if [IdRequest] { mutate { replace => { "IdRequest" => "%{[IdRequest][0]}" } } }

mutate {
remove_field => ["message","host", "http_poller_metadata"]
}

}

But i want to get a diferent doc per each severity record. Like:

logstash_testdiego | {
logstash_testdiego | "@version" => "1",
logstash_testdiego | "@timestamp" => 2019-04-16T09:24:37.860Z,
logstash_testdiego | "severity" => [
logstash_testdiego | [ 0] "info"
logstash_testdiego | }

logstash_testdiego | {
logstash_testdiego | "@version" => "1",
logstash_testdiego | "@timestamp" => 2019-04-16T09:24:37.860Z,
logstash_testdiego | "severity" => [
logstash_testdiego | [ 1] "info"
logstash_testdiego | }

....


#9

Once you have an array called severity you can us a split filter to create multiple events

split { field => "severity" }

(Diego Gutierrez) #10

yes its work. But i need something more.
Because I've more one field to split.

my XML config is:

xml {
    source => "message"
    remove_namespaces => true
    store_xml => false
    target => "alarmas"
    xpath => [
            "//severity/text()", "severity",
            "//alarmName/text()", "alarmName"
  ] 

 }

So i've need diferent doc with all fields:
my input xml is:

<SOAP:Envelope xmlns:SOAP="http://schemas.xmlsoap.org/soap/envelope/">
    <SOAP:Header>
        <header xmlns="xmlapi_1.0">
            <requestID>samostats@954345</requestID>
            <requestTime>Apr 17, 2019 11:08:19 AM</requestTime>
            <responseTime>Apr 17, 2019 11:08:19 AM</responseTime>
        </header>
    </SOAP:Header>
    <SOAP:Body>
        <findResponse xmlns="xmlapi_1.0">
            <result>
                <fm.AlarmObject>
                    <severity>info</severity>
                    <probableCause>260</probableCause>
                    <alarmName>363</alarmName>
                    <type>38</type>
                    <lastTimeDetected>1555352020018</lastTimeDetected>
                    <nodeName>LSTORRELODJA0</nodeName>
                    <alarmClassTag>radiusaccounting.RadiusAcctPlcyFailure</alarmClassTag>
                    <children-Set></children-Set>
                </fm.AlarmObject>
                <fm.AlarmObject>
                    <severity>info</severity>
                    <probableCause>89</probableCause>
                    <alarmName>8152</alarmName>
                    <type>4</type>
                    <lastTimeDetected>1549357659055</lastTimeDetected>
                    <nodeName>LSCANDELARGEN2</nodeName>
                    <alarmClassTag>netw.FileTransferFailure</alarmClassTag>
                    <children-Set></children-Set>
                </fm.AlarmObject>
                <fm.AlarmObject>
                    <severity>info</severity>
                    <probableCause>400</probableCause>
                    <alarmName>528</alarmName>
                    <type>45</type>
                    <lastTimeDetected>1543426495541</lastTimeDetected>
                    <nodeName>lssamma4</nodeName>
                    <alarmClassTag>schedule.scheduledTaskCompletionStatus</alarmClassTag>
                    <children-Set></children-Set>
                </fm.AlarmObject>
            </result>
        </findResponse>
    </SOAP:Body>
</SOAP:Envelope>

I want a doc per each <fm.AlarmObject>
so i expect

{
"@version" =&gt; "1",
"@timestamp" =&gt; 2019-04-15T13:14:05.431Z,
"severity"=&gt;"info"
"alarmName" =&gt;"363"
.......
},
{
"@version" =&gt; "1",
"@timestamp" =&gt; 2019-04-15T13:14:05.431Z,
"severity"=&gt;"info"
"alarmName" =&gt;"8152"
.......
},
{
"@version" =&gt; "1",
"@timestamp" =&gt; 2019-04-15T13:14:05.431Z,
"severity"=&gt;"info"
"alarmName" =&gt;"528"
.......
}

but if I use:

split { field => "severity" }
split { field => "alarmName" }

my output is:

 {
logstash_testdiego |      "alarmName" => "363",
logstash_testdiego |     "@timestamp" => 2019-04-17T10:41:36.451Z,
logstash_testdiego |           "path" => "/usr/share/logstash/datos/gestores/file.xml",
logstash_testdiego |       "severity" => "info",
logstash_testdiego |       "@version" => "1"
logstash_testdiego | }
logstash_testdiego | {
logstash_testdiego |      "alarmName" => "8152",
logstash_testdiego |     "@timestamp" => 2019-04-17T10:41:36.451Z,
logstash_testdiego |           "path" => "/usr/share/logstash/datos/gestores/file.xml",
logstash_testdiego |       "severity" => "info",
logstash_testdiego |       "@version" => "1"
logstash_testdiego | }
logstash_testdiego | {
logstash_testdiego |      "alarmName" => "528",
logstash_testdiego |     "@timestamp" => 2019-04-17T10:41:36.451Z,
logstash_testdiego |           "path" => "/usr/share/logstash/datos/gestores/file.xml",
logstash_testdiego |       "severity" => "info",
logstash_testdiego |       "@version" => "1"
logstash_testdiego | }
logstash_testdiego | {
logstash_testdiego |      "alarmName" => "363",
logstash_testdiego |     "@timestamp" => 2019-04-17T10:41:36.451Z,
logstash_testdiego |           "path" => "/usr/share/logstash/datos/gestores/file.xml",
logstash_testdiego |       "severity" => "info",
logstash_testdiego |       "@version" => "1"
logstash_testdiego | }
logstash_testdiego | {
logstash_testdiego |      "alarmName" => "8152",
logstash_testdiego |     "@timestamp" => 2019-04-17T10:41:36.451Z,
logstash_testdiego |           "path" => "/usr/share/logstash/datos/gestores/file.xml",
logstash_testdiego |       "severity" => "info",
logstash_testdiego |       "@version" => "1"
logstash_testdiego | }
logstash_testdiego | {
logstash_testdiego |      "alarmName" => "528",
logstash_testdiego |     "@timestamp" => 2019-04-17T10:41:36.451Z,
logstash_testdiego |           "path" => "/usr/share/logstash/datos/gestores/file.xml",
logstash_testdiego |       "severity" => "info",
logstash_testdiego |       "@version" => "1"
logstash_testdiego | }
logstash_testdiego | {
logstash_testdiego |      "alarmName" => "363",
logstash_testdiego |     "@timestamp" => 2019-04-17T10:41:36.451Z,
logstash_testdiego |           "path" => "/usr/share/logstash/datos/gestores/file.xml",
logstash_testdiego |       "severity" => "info",
logstash_testdiego |       "@version" => "1"
logstash_testdiego | }
logstash_testdiego | {
logstash_testdiego |      "alarmName" => "8152",
logstash_testdiego |     "@timestamp" => 2019-04-17T10:41:36.451Z,
logstash_testdiego |           "path" => "/usr/share/logstash/datos/gestores/file.xml",
logstash_testdiego |       "severity" => "info",
logstash_testdiego |       "@version" => "1"
logstash_testdiego | }
logstash_testdiego | {
logstash_testdiego |      "alarmName" => "528",
logstash_testdiego |     "@timestamp" => 2019-04-17T10:41:36.451Z,
logstash_testdiego |           "path" => "/usr/share/logstash/datos/gestores/file.xml",
logstash_testdiego |       "severity" => "info",
logstash_testdiego |       "@version" => "1"
logstash_testdiego | }

I expect only 3 docs but i've received a combination per each field with the other. And i cant create a array with split :frowning:


#11

Your post is unreadable. You XML is being interpreted as HTML tags so none of the elements actually appear in the post. Edit your posts and either select the XML (or configuration) and click on </> in the toolbar above the edit pane to treat it as a block quote, or surround the lines of XML with lines containing just three backticks...

```

(Diego Gutierrez) #12

Sorry is my first post in this discuss.
Now is better?


#13

Much better. Does

    xml {
        source => "message"
        remove_namespaces => true
        store_xml => true
        target => "alarmas"
        force_array => false
        remove_field => [ "message" ]
    }
    split { field => "[alarmas][Body][findResponse][result][fm.AlarmObject]" }

work for you?


(Diego Gutierrez) #14

I got it with the following way:

filter{
  ## interpret the message payload as XML
  xml {
    source => "message"
    remove_namespaces => true
    store_xml => false
    target => "alarmas"
    xpath => {"//fm.AlarmObject" => "alarma"}

 }

  split { field => "alarma"}

  mutate {
  gsub=>["alarma", "\n", ""]
  # split=>{"alarma" => "\n"}

  remove_field => ["message","host", "http_poller_metadata","alarmas", "tags"] 
  }

  xml {
    source => "alarma"
    remove_namespaces => true
    store_xml => false
    # target => "alarms"
    xpath => [
      "//severity/text()" , "severity",
      "//probableCause/text()" , "probableCause",
      "//alarmName/text()" , "alarmName",
      "//type/text()" , "type",
      "//alarmClassTag/text()" , "alarmClassTag",
      "//nodeName/text()" , "NodeName"
    ]

 }

 mutate{
   remove_field => ["alarma"]
 }
 mutate { replace => [ 
   "severity", "%{[severity][0]}",
   "probableCause", "%{[probableCause][0]}",
   "alarmName", "%{[alarmName][0]}",
   "type", "%{[type][0]}",
   "alarmClassTag", "%{[alarmClassTag][0]}",
   "NodeName","%{[NodeName][0]}"
   ] } 

}

maybe my way is not the best to the processing speed but i get the result like i want it:

logstash_testdiego | {
logstash_testdiego |              "type" => "38",
logstash_testdiego |     "alarmClassTag" => "radiusaccounting.RadiusAcctPlcyFailure",
logstash_testdiego |        "@timestamp" => 2019-04-17T14:11:48.664Z,
logstash_testdiego |          "@version" => "1",
logstash_testdiego |          "severity" => "info",
logstash_testdiego |     "probableCause" => "260",
logstash_testdiego |         "alarmName" => "363",
logstash_testdiego |          "NodeName" => "LSTORRELODJA0",
logstash_testdiego |              "path" => "/usr/share/logstash/datos/gestores/file.xml"
logstash_testdiego | }
logstash_testdiego | {
logstash_testdiego |              "type" => "4",
logstash_testdiego |     "alarmClassTag" => "netw.FileTransferFailure",
logstash_testdiego |        "@timestamp" => 2019-04-17T14:11:48.664Z,
logstash_testdiego |          "@version" => "1",
logstash_testdiego |          "severity" => "info",
logstash_testdiego |     "probableCause" => "89",
logstash_testdiego |         "alarmName" => "8152",
logstash_testdiego |          "NodeName" => "LSCANDELARGEN2",
logstash_testdiego |              "path" => "/usr/share/logstash/datos/gestores/file.xml"
logstash_testdiego | }
logstash_testdiego | {
logstash_testdiego |              "type" => "45",
logstash_testdiego |     "alarmClassTag" => "schedule.scheduledTaskCompletionStatus",
logstash_testdiego |        "@timestamp" => 2019-04-17T14:11:48.664Z,
logstash_testdiego |          "@version" => "1",
logstash_testdiego |          "severity" => "info",
logstash_testdiego |     "probableCause" => "400",
logstash_testdiego |         "alarmName" => "528",
logstash_testdiego |          "NodeName" => "lssamma4",
logstash_testdiego |              "path" => "/usr/share/logstash/datos/gestores/file.xml"
logstash_testdiego | }

anyway i will try with ur solution. Thanks so much to the patience with me jejejejej