XML Filter example

Elastic Stack Logstash
1

Hi,

I have a requirement where in the a XML message is picked up from TIBCO EMS with the help of logstash and then is viewed in Kibana. I want to parse this message element individually for example conversationid,eventid etc. to be viewed in kibana. can anyone help regarding this.
Want to parse the XML received from TIBCO EMS mentioned below.

<tns:ActivityInput xmlns:tns="http://www.tibco.com/namespaces/tnt/plugins/jms+32553768-9d2e-4ed0-90ae-0a1e20803547+input" xmlns:tns2="http://www.ericsson.com/tibco/schema/Logger">
  <Body>
    <tns2:Logger_Request>
      <tns2:conversationId>TIB-22b0be25-9e87-49ba-a3c0-1b617c82212a</tns2:conversationId>
      <tns2:correlationId>4618bdb4-36fb-4a37-b161-7fb07aeaedd6</tns2:correlationId>
      <tns2:eventId>CUST0000000023</tns2:eventId>
      <tns2:logTimestamp>2021-02-17T17:49:39.828+05:30</tns2:logTimestamp>
      <tns2:type>START</tns2:type>
      <tns2:businessReferenceId>228526</tns2:businessReferenceId>
      <tns2:systemConsumer>DFE</tns2:systemConsumer>
      <tns2:systemDestination>EB</tns2:systemDestination>
      <tns2:serviceName>SubmitPayment</tns2:serviceName>
      <tns2:operationName>POST</tns2:operationName>
      <tns2:payload>&lt;?xml version="1.0" encoding="UTF-8"?&gt;&lt;CustomerPayment xmlns="http://xmlns.ericsson.com/CDM/Payment" xmlns:ns1="http://xmlns.ericsson.com/CDM/Base"&gt;&lt;ns1:customerId&gt;CUST0000000023&lt;/ns1:customerId&gt;&lt;ns1:billingAccountId&gt;BA0000000044&lt;/ns1:billingAccountId&gt;&lt;transactionType&gt;DEPOSIT12&lt;/transactionType&gt;&lt;transactionRefNum&gt;228526&lt;/transactionRefNum&gt;&lt;Amount&gt;&lt;ns1:amount&gt;1.0E16&lt;/ns1:amount&gt;&lt;ns1:units&gt;&lt;ns1:currencyCode&gt;IDR&lt;/ns1:currencyCode&gt;&lt;/ns1:units&gt;&lt;/Amount&gt;&lt;referenceDate&gt;2020-12-21&lt;/referenceDate&gt;&lt;entryDate&gt;2021-02-11&lt;/entryDate&gt;&lt;/CustomerPayment&gt;</tns2:payload>
      <tns2:Log-Level>INFO</tns2:Log-Level>
      <tns2:appSpace>BWEclipseAppSpace</tns2:appSpace>
      <tns2:appNode>BWEclipseAppNode</tns2:appNode>
      <tns2:engine>Main</tns2:engine>
      <tns2:appModule>GW.SubmitPayment</tns2:appModule>
    </tns2:Logger_Request>
  </Body>
</tns:ActivityInput>
2

Have you tried using an xml filter? If so, what did you use and what issues did you have with the result?

3

Hi Badger,

I am very new to logstash and by looking at various blogs i have applied the following filter. But it didn't worked

filter
{
        xml {
        remove_namespaces => "true"
        source => "message"
        store_xml => "true"
        target => "doc"
        force_array => false
        xpath => ["//conversationId/text()","conversationId"]
    }

}
4

With that XML and that filter configuration I get

"conversationId" => "TIB-22b0be25-9e87-49ba-a3c0-1b617c82212a",
           "doc" => {
     "xmlns:tns" => "http://www.tibco.com/namespaces/tnt/plugins/jms+32553768-9d2e-4ed0-90ae-0a1e20803547+input",
    "xmlns:tns2" => "http://www.ericsson.com/tibco/schema/Logger",
          "Body" => {
        "Logger_Request" => {
                        "eventId" => "CUST0000000023",
                      "Log-Level" => "INFO",
                 "conversationId" => "TIB-22b0be25-9e87-49ba-a3c0-1b617c82212a",
                        "appNode" => "BWEclipseAppNode",
...

So what exactly do you mean by "it didn't work"?

1 Like
5

Hi Badger,

Sorry it is working. The XML node i copied and pasted was incorrect. My bad


input {
   jms {
      include_header => true
      include_properties => true
      include_body => true
      use_jms_timestamp => true
      timeout => -1
      destination => "q.logstash.command"
      pub_sub => false
      yaml_file => "/home/elastic/softwares/logstash-7.11.1/config/ems.yml"
      yaml_section => "ems"
   }
}

filter
{
 xml {
        remove_namespaces => "true"
        source => "message"
        store_xml => "false"
        target => "doc"
        xpath =>  [
                   "//conversationId/text()","conversationId",
                   "//eventId/text()","eventId",
                   "//correlationId/text()","correlationId",
                   "//systemConsumer/text()","systemConsumer",
                   "//systemDestination/text()","systemDestination",
                   "//serviceName/text()","serviceName",
                   "//payload/text()","payload"
        ]

    }
}

output{
        elasticsearch {
        hosts => ["localhost:9200"]
        index => "inderforlogstash"
        }
}

Thanks,
Rabin

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.