# Get the XML node from unformatted xml

**URL:** <https://discuss.elastic.co/t/get-the-xml-node-from-unformatted-xml/163576>\
**Category:** Logstash\
**Created:** [January 9, 2019, 4:01pm UTC](https://discuss.elastic.co/t/get-the-xml-node-from-unformatted-xml/163576 "2019-01-09T16:01:35Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![rainman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rainman/32/43218_2.png) [@rainman](https://discuss.elastic.co/u/rainman)\
**Post date:** [January 9, 2019, 4:01pm UTC](https://discuss.elastic.co/t/get-the-xml-node-from-unformatted-xml/163576/1 "2019-01-09T16:01:35Z")

</div>

I have a xml that is not parsing correctly unless I prettify the xml. XML tags can be a new line or same line with the last end tag, and does not have indentation for it. but i tried with prettify it worked fine. can we achieve this with out prettify or do we need to prettify the xml befor we process with logstash? Is there a way that i can handle with logstash?

**Below is my xml**

\<?xml version="1.0" encoding="utf-8"?\>\<batch\_upload\>\<batches\>\<batch\>\<id\>00010\</id\>\<title\>Batch Title\</title\>\<description\>\<![CDATA[We're one of the largest Membership organizations in the country, but we’re so much more than our legendary roadside service. We call our club's vision, mission, values, and supporting pillars "Our House" because they are the foundation for all that we do.&nbsp; We're working to transform life by unleashing the innovative spirit of our Team Members. We're community minded, and celebrate the growth, development and successes of our diverse Team Members. .]]\>\</description\>\<city\>Anchorage\</city\>\<state\>AK\</state\>\<zipcode\>99503\</zipcode\>\<country\>USA\</country\>\<parameters /\>\<groups\>\<group type="1"\>10\</group\>\<group type="1"\>11\</group\>\<classification type="1"\>12\</group\>\<group type="2" /\>212\</group\>\<requirements /\>\<person\>\<name\>Person one\</name\>\<methods\>\<method type="online"\>Method 1\</method\>\</methods\>\</person\>\</batch\>\</batches\>\</batch\_upload\>

**below is my logstash config**

input {  
file {  
path =\> "batches.xml"  
start\_position =\> beginning  
sincedb\_path =\> "NUL"  
type =\> "xml"  
codec =\> multiline  
{  
pattern =\> "|\</batches"  
negate =\> true  
what =\> "previous"  
auto\_flush\_interval =\> 1  
}  
}  
}

filter {  
if [message] == "" or [message] == "" or [message] == "\r" {  
drop {}  
}  
xml {  
store\_xml =\> false  
source =\> "message"  
target =\> "message.parsed"  
xpath =\> [  
"/batch/id/text()", batch\_id,  
"/batch/title/text()", batch\_title  
]  
force\_array =\> false  
}  
}

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 9, 2019, 4:22pm UTC](https://discuss.elastic.co/t/get-the-xml-node-from-unformatted-xml/163576/2 "2019-01-09T16:22:23Z")

</div>

You do not need to prettify XML before parsing it. Please edit your post and use \</\> in the tool bar above the edit pane to preserve the formatting of your XML and configuration.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 9, 2019, 7:00pm UTC](https://discuss.elastic.co/t/get-the-xml-node-from-unformatted-xml/163576/3 "2019-01-09T19:00:30Z")

</div>

> [@rainman](#):
>
> \<groups\>\<group type="1"\>10\</group\>\<group type="1"\>11\</group\>\<classification type="1"\>12\</group\>\<group type="2" /\>212\</group\>

That is not valid XML. The groups element is never closed. The final group is closed using both /\> and \</group\>. And the classification element is also closed using \</group\>. You cannot have spaces in \<requirements /\> (likewise parameters).

Once you fix all that you have to have the complete path in the xpath.

```
    xpath => {
        "/batch_upload/batches/batch/id/text()" => batch_id
        "/batch_upload/batches/batch/title/text()" => batch_title
    }

```

---

<div class="post-metadata">

**Author:** ![rainman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rainman/32/43218_2.png) [@rainman](https://discuss.elastic.co/u/rainman)\
**Post date:** [January 9, 2019, 7:42pm UTC](https://discuss.elastic.co/t/get-the-xml-node-from-unformatted-xml/163576/4 "2019-01-09T19:42:46Z")

</div>

Thank You Badger for the reply,

its a typo.

**here is the original XML we get.**

\<?xml version="1.0" encoding="utf-8"?\>\<batch\_upload\>\<batches\>\<batch\>\<id\>0101\</id\>\<title\>Some Title 1\</title\>\<description\>\<![CDATA[We're one of the largest Membership organizations in the country, but we’re so much more than our legendary roadside service. We call our club's vision, mission, values, and supporting pillars "Our House" because they are the foundation for all that we do. We're working to transform AAA for the next century with a mission to create Members

- Sells International & Domestic vacation packages, cruises, tours, hotel, car rental, rail and air travel

- Owns the relationship with the member from beginning to end of each travel related transaction

- Researches, evaluates and compares appropriate AAA Travel Partner packages to match up with member needs for the purpose of “delivering exceptional member experiences” in every transaction

for life by unleashing the innovative spirit of our Team Members. We're community minded, and celebrate the growth, development and successes of our diverse Team Members.]]\>\</description\>\<city\>Anchorage\</city\>\<state\>AK\</state\>\<zipcode\>99503\</zipcode\>\<country\>USA\</country\>\<dateacquired\>2018-12-19T08:19:40-05:00\</dateacquired\>\<parameters /\>\<groups\>\<group type="1"\>222\</group\>\<group type="1"\>333\</group\>\<group type="1"\>444\</group\>\<group type="2" /\>\</groups\>\<requirements /\>\<application\>\<person\>Person One\</person\>\<methods\>\<method type="online"\>5252klg\</method\>\</methods\>\</application\>\</batch\>\<batch\>\<id\>0202\</id\>\<title\>Some Title 2\</title\>\<description\>\<![CDATA[We're one of the largest Membership organizations in the country, but we’re so much more than our legendary roadside service. We call our club's vision, mission, values, and supporting pillars

- Sells International & Domestic vacation packages, cruises, tours, hotel, car rental, rail and air travel

- Owns the relationship with the member from beginning to end of each travel related transaction

- Researches, evaluates and compares appropriate AAA Travel Partner packages to match up with member needs for the purpose of “delivering exceptional member experiences” in every transaction

"Our House" because they are the foundation for all that we do. We're working to transform AAA for the next century with a mission to create Members for life by unleashing the innovative spirit of our Team Members.]]\>\</description\>\<city\>Anchorage\</city\>\<state\>AK\</state\>\<zipcode\>99503\</zipcode\>\<country\>USA\</country\>\<dateacquired\>2018-12-19T23:30:11-05:00\</dateacquired\>\<parameters /\>\<groups\>\<group type="1"\>545454\</group\>\<group type="1"\>4545\</group\>\<group type="1"\>7878\</group\>\<group type="2" /\>\</groups\>\<requirements /\>\<application\>\<person\>Person two\</person\>\<methods\>\<method type="online"\>213234sdf\</method\>\</methods\>\</application\>\</batch\>\<batch\>\<id\>0303\</id\>\<title\>Some Title 3\</title\>\<description\>\<![CDATA[We are s -

- Sells International & Domestic vacation packages, cruises, tours, hotel, car rental, rail and air travel

- Owns the relationship with the member from beginning to end of each travel related transaction

- Researches, evaluates and compares appropriate AAA Travel Partner packages to match up with member needs for the purpose of “delivering exceptional member experiences” in every transaction

an industry leader in the sales and lease-to-own retailer known for quality brand names and superior customer service. We provide our team members the opportunity to reach their full potential in a team-oriented, high-energy, recognition-based environment with competitive pay and benefits. This is much more than a batch – It is a career with purpose]]\>\</description\>\<city\>Anchorage\</city\>\<state\>AK\</state\>\<zipcode\>99503\</zipcode\>\<country\>USA\</country\>\<dateacquired\>2018-12-05T03:13:44-05:00\</dateacquired\>\<parameters /\>\<groups\>\<group type="1"\>454545\</group\>\<group type="1"\>7778\</group\>\<group type="1"\>45555\</group\>\<group type="2" /\>\</groups\>\<requirements /\>\<application\>\<person\>Person three\</person\>\<methods\>\<method type="online"\>asdfsafwer23\</method\>\</methods\>\</application\>\</batch\>\</batches\>\</batch\_upload\>

---

<div class="post-metadata">

**Author:** ![rainman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rainman/32/43218_2.png) [@rainman](https://discuss.elastic.co/u/rainman)\
**Post date:** [January 9, 2019, 7:46pm UTC](https://discuss.elastic.co/t/get-the-xml-node-from-unformatted-xml/163576/5 "2019-01-09T19:46:48Z")

</div>

Thanks again Badger, I appreciate your help.

I want you know that we do have multiple batches. and the file is huge. can you help reading each batch a different event not the whole batches in single event.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 9, 2019, 10:09pm UTC](https://discuss.elastic.co/t/get-the-xml-node-from-unformatted-xml/163576/6 "2019-01-09T22:09:56Z")

</div>

OK, so this is less about XML and more about how to create events using a file input. There are a couple of options.

You say the file is huge, but do not quantify that. Huge is different things in different circumstances. For example, if you have a 64-bit JVM on a box with 2 TB of memory you could possibly ingest 100 GB files as single events. I'd say if the file size is more than 10% of the heap it is unlikely to work.

If you are going to use a multiline codec to handle each batch then you need to prettify it enough for the start of a batch to be on a new line, so that you can use a pattern that matches the end of a batch.

Then you will need some mutate+gsub to get rid of the xml and batch\_upload elements.

Does a batch\_upload element ever contain more than one batches element?

---

<div class="post-metadata">

**Author:** ![rainman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rainman/32/43218_2.png) [@rainman](https://discuss.elastic.co/u/rainman)\
**Post date:** [January 17, 2019, 4:38pm UTC](https://discuss.elastic.co/t/get-the-xml-node-from-unformatted-xml/163576/7 "2019-01-17T16:38:06Z")

</div>

Thank You, Badger.

I was very helpful!  
I am working on the xml file will get back to you if I see any obstacle.

---

<div class="post-metadata">

**Author:** ![rainman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rainman/32/43218_2.png) [@rainman](https://discuss.elastic.co/u/rainman)\
**Post date:** [January 25, 2019, 10:00pm UTC](https://discuss.elastic.co/t/get-the-xml-node-from-unformatted-xml/163576/8 "2019-01-25T22:00:18Z")

</div>

As you suggested we are formatting the XML file before processing the with Logstash. Below is the process we are following and found the documents processed to ES is holding the different number of the document on every run.

1. We did validate the XML.

2. We got the count of the **Batches** to make sure the documents are processed completely.

3. FYI, the file we are processing is holding **1849702 batches.**

4. First time I got **1849840** , Next time with a different number...

5. We formatted the XML element to make sure every XML is starting in New Line i.e. We are adding Environment.NewLine with C# if we peek into it this is how the output is holding (\r\n) before every XML element.

6. Below is my logstash config

**Input**  
**Please do check the max\_lines property.** we need this because object descriptions lines can be between 7000 to 10000 on an average.

input {  
file {  
path =\> "batches.xml"  
start\_position =\> beginning  
sincedb\_path =\> "NUL"  
type =\> "xml"  
codec =\> multiline  
{  
pattern =\>`"<batch>|<batch>\n|<batch>\r\n|<batch>\r"`  
negate =\> true  
what =\> "previous"  
max\_lines =\> 10000  
auto\_flush\_interval =\> 1  
}  
}  
}

**Filters**

filter {

if [message] == `"<batches>"` or [message] == `"</batches>"` or [message] =~ `"<?xml version"` or [message] =~ `"batches>"` or [message] =~ `"batch_upload>"` {  
drop {}  
}  
xml {  
store\_xml =\> false  
source =\> "message"  
target =\> "message.parsed"  
xpath =\> [  
"/job/id/text()", batch\_id,  
"/job/title/text()", batch\_title,  
"/job/zipcode/text()", zipcode,  
"/job/application/country/text()", country  
]  
force\_array =\> false  
}

mutate {  
remove\_field =\> ["path","host","type","tags"]  
}  
fingerprint {  
target =\> "uuid"  
method =\> "UUID"  
}  
}

**Output**

output {  
elasticsearch {  
index =\> "batchesindex"  
document\_type =\> "batches"  
hosts =\> "10.10.10.3:9200"  
manage\_template =\> true  
template =\> "/data/test/config/elasticsearch-template1.json"  
template\_overwrite =\> "true"  
document\_id =\> "%{@timestamp}%{uuid}"  
}  
}

**By using this configuration**

I can accomplish to load the complete batches without any issue if I brake it down to the small file, that is holding 20 to 30 **batches** , we are following the above steps and using the same config

But when I try to load the whole file is see some events are not processed properly.  
i.e.

Message is holding from the start tag of **`<id>00010</id>`**... In this case we are having extra batches documents been processed to ES.

Can you please help us with this issue. How to make sure the batches are processed **completely** without any **data leeks or duplicates** or **event brake downs.**

---

<div class="post-metadata">

**Author:** ![rainman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rainman/32/43218_2.png) [@rainman](https://discuss.elastic.co/u/rainman)\
**Post date:** [January 30, 2019, 3:41pm UTC](https://discuss.elastic.co/t/get-the-xml-node-from-unformatted-xml/163576/9 "2019-01-30T15:41:49Z")

</div>

@Badger/ @magnusbaeck Can you please suggest on this!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 27, 2019, 3:41pm UTC](https://discuss.elastic.co/t/get-the-xml-node-from-unformatted-xml/163576/10 "2019-02-27T15:41:54Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.