# Logstash xml parsing - Fields extraction

**URL:** <https://discuss.elastic.co/t/logstash-xml-parsing-fields-extraction/123201>\
**Category:** Logstash\
**Created:** [March 9, 2018, 6:20am UTC](https://discuss.elastic.co/t/logstash-xml-parsing-fields-extraction/123201 "2018-03-09T06:20:15Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![satkumvnr](https://avatars.discourse-cdn.com/v4/letter/s/50afbb/32.png) [@satkumvnr](https://discuss.elastic.co/u/satkumvnr)\
**Post date:** [March 9, 2018, 6:20am UTC](https://discuss.elastic.co/t/logstash-xml-parsing-fields-extraction/123201/1 "2018-03-09T06:20:15Z")

</div>

HI,  
I am new to Elastic Search  
I am streaming the xml files from Apache Kafka to elastic search, where I need to assign the fields for xml attributes. I am struggling to get pass through it. I need to get the below fields assigned, so that I can view it from Kibana  
eventCreationDtm  
eventCreationDtmStr  
tagIssDtm  
bagOrigArpt  
destArptCd

\<?xml version="1.0" encoding="UTF-8"?\>

\<ns0:Envelope xmlns:ns0="[http://www.test.com/Schema/BAG.xsd](http://www.test.com/Schema/BAG.xsd)"\>  
\<ns1:eventHeader xmlns:ns1="[http://www.test.com/eai/event/header](http://www.test.com/eai/event/header)" eventName="BGIC" eventCreationSys="Baggage" eventCreationDtm="2018-02-24T21:33:03.698Z" eventActionCd="BGIC" eventID="4016426319" version="2.0.0"\>  
\<ns1:srcSys eventID="4016426319" eventName="BGIC" eventCreationSys="Baggage" eventCreationDtmStr="2018-02-24T21:33:03.698Z" processLoc=“IXM”\>  
ns1:usr/  
\</ns1:srcSys\>  
\</ns1:eventHeader\>  
ns0:Body  
  
  
FRT  
BHK  
B  
99  
false  
false  
false  
false  
H3HHSH  
  
  
LASTNAME  
FIRSTNAME  
9JSIJF8  
  
  
  
true

I have added the sample xml. Pls assist on this,

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 9, 2018, 7:14am UTC](https://discuss.elastic.co/t/logstash-xml-parsing-fields-extraction/123201/2 "2018-03-09T07:14:35Z")

</div>

Make sure to format XML as preformatted text with the `</>` toolbar button. As you can see your XML has been mangled.

Have you looked at the xml filter? Its xpath option should make it very easy to extract the contents of the elements/attributes that you list.

---

<div class="post-metadata">

**Author:** ![satkumvnr](https://avatars.discourse-cdn.com/v4/letter/s/50afbb/32.png) [@satkumvnr](https://discuss.elastic.co/u/satkumvnr)\
**Post date:** [March 12, 2018, 11:00am UTC](https://discuss.elastic.co/t/logstash-xml-parsing-fields-extraction/123201/3 "2018-03-12T11:00:30Z")

</div>

HI Magnus, I have the below sample xml`\<?xml version="1.0" encoding="UTF-8"?\>

\<ns0:Envelope xmlns:ns0="[http://www.TEST.com/Schema/BAG.xsd](http://www.TEST.com/Schema/BAG.xsd)"\>  
\<ns1:eventHeader xmlns:ns1="[http://www.TEST.com/eai/event/header](http://www.TEST.com/eai/event/header)" eventName="JKJK" eventCreationSys="Baggage" eventCreationDtm="2018-02-24T21:33:03.698Z" eventActionCd="JKJK" eventID="11111" version="2.0.0"\>  
\<ns1:srcSys eventID="11111" eventName="TEST" eventCreationSys="Baggage" eventCreationDtmStr="2018-02-24T21:33:03.698Z" processLoc="TEST"\>  
ns1:usr/  
\</ns1:srcSys\>  
\</ns1:eventHeader\>  
ns0:Body  
  
  
AAA  
BBB  
B  
3  
false  
false  
false  
false  
3FERR33  
  
  
\</ns0:Body\>  
\</ns0:Envelope\>`

The logstash.conf looks like  
`input {  
kafka {  
bootstrap\_servers =\> "localhost:9092"  
topics =\> ["Hello1"]  
}  
}

filter {  
xml {  
store\_xml =\> false  
source =\> "message"  
xpath =\> ["/ns0:Envelope/ns0:Body/bagDetails/@tagIssDtm/string()", "date"]  
}

date {  
match =\> ["date" , "dd-MM-yyyy HH:mm:ss"]  
timezone =\> "Europe/Amsterdam"  
}  
}

output {  
stdout {codec=\> "json\_lines"}  
elasticsearch {  
hosts =\> ["localhost:9200"]  
index =\> "elasticse"  
}  
}`

I am testing for one attribute and getting exception. I need to get the below attributes,  
tagIssDtm  
primaryTypePriority  
eventCreationDtm

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 12, 2018, 11:03am UTC](https://discuss.elastic.co/t/logstash-xml-parsing-fields-extraction/123201/4 "2018-03-12T11:03:44Z")

</div>

Your XML is still mangled because you didn't format your post as requested.

---

<div class="post-metadata">

**Author:** ![satkumvnr](https://avatars.discourse-cdn.com/v4/letter/s/50afbb/32.png) [@satkumvnr](https://discuss.elastic.co/u/satkumvnr)\
**Post date:** [March 12, 2018, 11:44am UTC](https://discuss.elastic.co/t/logstash-xml-parsing-fields-extraction/123201/5 "2018-03-12T11:44:40Z")

</div>

Again pasted the xml below,

 \<?xml version="1.0" encoding="UTF-8"?\>

```
<ns0:Envelope xmlns:ns0="http://www.TEST.com/Schema/BAG.xsd">
    <ns1:eventHeader xmlns:ns1="http://www.TEST.com/eai/event/header" eventName="JKJK" eventCreationSys="Baggage" eventCreationDtm="2018-02-24T21:33:03.698Z" eventActionCd="JKJK" eventID="11111" version="2.0.0">
        <ns1:srcSys eventID="11111" eventName="TEST" eventCreationSys="Baggage" eventCreationDtmStr="2018-02-24T21:33:03.698Z" processLoc="TEST">
            <ns1:usr/>
        </ns1:srcSys>
    </ns1:eventHeader>
    <ns0:Body>
        <bagDetails tagNbr="11111" tagUniqKey="22222" tagIssDtm="2018-02-24T19:03:49.368Z" bagTagActvInd="true">
            <bagInfo>
                <bagOrigArpt>AAA</bagOrigArpt>
                <bagTermArpt>BBB</bagTermArpt>
                <tagPrimaryType>B</tagPrimaryType>
                <primaryTypePriority>3</primaryTypePriority>
                <isPriority>false</isPriority>
                <isHeavy>false</isHeavy>
                <isRush>false</isRush>
                <isSelectee>false</isSelectee>
                <printerId>3FERR33</printerId>
            </bagInfo>
        </bagDetails>
    </ns0:Body>
</ns0:Envelope>
```

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 12, 2018, 9:40pm UTC](https://discuss.elastic.co/t/logstash-xml-parsing-fields-extraction/123201/6 "2018-03-12T21:40:30Z")

</div>

Okay. What you have should work, but I recall there being some problems with XML namespaces. Have you tried enableing the `remove_namespaces` option and changing the XPath expression to /Envelope/Body/bagDetails/@tagIssDtm/string()?

---

<div class="post-metadata">

**Author:** ![satkumvnr](https://avatars.discourse-cdn.com/v4/letter/s/50afbb/32.png) [@satkumvnr](https://discuss.elastic.co/u/satkumvnr)\
**Post date:** [March 13, 2018, 4:53am UTC](https://discuss.elastic.co/t/logstash-xml-parsing-fields-extraction/123201/7 "2018-03-13T04:53:19Z")

</div>

HI Magnus,

I tried with the below conf and still the syntax error occurs.

```
   filter {
  xml {
   store_xml => false
   source => "message"
   remove_namespaces => "true"
   xpath => ["/Envelope/Body/bagDetails/@tagIssDtm/string()", "date"]
      }
 
date {
    match => ["date" , "dd-MM-yyyy HH:mm:ss"]
    timezone => "Europe/Amsterdam"
     }
     } 

```

Errors,  
[2018-03-13T10:25:24,879][ERROR][logstash.pipeline] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {:pipeline\_id=\>"main", "exception"=\>"/Envelope/Body/bagDetails/@tagIssDtm/string()", "backtrace"=\>["nokogiri/XmlXpathContext.java:130:in `evaluate'", "/Users/sathish/apps/logstash/vendor/bundle/jruby/2.3.0/gems/nokogiri-1.8.2-java/lib/nokogiri/xml/searchable.rb:198:in`xpath\_impl'"

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [March 13, 2018, 6:47am UTC](https://discuss.elastic.co/t/logstash-xml-parsing-fields-extraction/123201/8 "2018-03-13T06:47:28Z")

</div>

Then I don't know what's going on.

---

<div class="post-metadata">

**Author:** ![satkumvnr](https://avatars.discourse-cdn.com/v4/letter/s/50afbb/32.png) [@satkumvnr](https://discuss.elastic.co/u/satkumvnr)\
**Post date:** [March 13, 2018, 7:04am UTC](https://discuss.elastic.co/t/logstash-xml-parsing-fields-extraction/123201/9 "2018-03-13T07:04:04Z")

</div>

remove\_namespaces is not removing the ns tag

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 10, 2018, 7:04am UTC](https://discuss.elastic.co/t/logstash-xml-parsing-fields-extraction/123201/10 "2018-04-10T07:04:10Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.