# Parsing multiline, netsted XML using Logstash

**URL:** <https://discuss.elastic.co/t/parsing-multiline-netsted-xml-using-logstash/172142>\
**Category:** Logstash\
**Created:** [March 13, 2019, 12:11pm UTC](https://discuss.elastic.co/t/parsing-multiline-netsted-xml-using-logstash/172142 "2019-03-13T12:11:39Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![saif3r](https://avatars.discourse-cdn.com/v4/letter/s/49beb7/32.png) [@saif3r](https://discuss.elastic.co/u/saif3r)\
**Post date:** [March 13, 2019, 12:11pm UTC](https://discuss.elastic.co/t/parsing-multiline-netsted-xml-using-logstash/172142/1 "2019-03-13T12:11:39Z")

</div>

Hello Guys,

I'm currently learning Logstash using one of Udemy's courses. While the course is currently focusing on file/http/stdin inputs, I would like to know whether it is possible to parse an xml presented below. This would help me asses the Logstash as one of the possible candidates for such operation.  
So the xml looks more or less like this:

```
<?xml version="1.0" encoding="UTF-8"?>
<BATCH TIMESTAMP="2019-01-02T04:04:51.931+01:00" SOFTWARE="1.0" HARDWARE="1.0">
<PLANT NAME="PARIS" LINE="PRODLINE1" TESTER="TESTER1"/>
<PRODUCT ID="3" NAME="MOTHERBOARD" STATUS="OK">
	<GROUP ID="1" NAME="TESTGROUP1">
		<TEST ID="10" NAME="VOLTAGETEST" VALUE="2.34523" STATUS="OK"/>
		<TEST ID="20" NAME="INTEGRATIONTEST" VALUE="1" STATUS="NOK"/>
	</GROUP>
	<GROUP ID="2" NAME="TESTGROUP2">
		<TEST ID="10" NAME="VISUALTEST" VALUE="22" STATUS="OK"/>
	</GROUP>
</PRODUCT>
</BATCH>

```

This is the output I would like to receive in Elasticsearch (3 records in total):

```
{"BATCH_TIMESTAMP": "2019-01-02T04:04:51.931+01:00", "BATCH_SOFTWARE": "1.0", "BATCH_HARDWARE": "1.0", "PLANT_LINE": "PRODLINE1", "PLANT_TESTER": "TESTER1" "PRODUCT_ID": "3", "PRODUCT_NAME": "MOTHERBOARD", "GROUP_ID": "1", "GROUP_NAME": "TESTERGROUP1", "TEST_ID": "10", "TEST_NAME": "VOLTAGETEST", "TEST_VALUE": "2.34523", "TEST_STATUS": "OK"}
{"BATCH_TIMESTAMP": "2019-01-02T04:04:51.931+01:00", "BATCH_SOFTWARE": "1.0", "BATCH_HARDWARE": "1.0", "PLANT_LINE": "PRODLINE1", "PLANT_TESTER": "TESTER1" "PRODUCT_ID": "3", "PRODUCT_NAME": "MOTHERBOARD", "GROUP_ID": "1", "GROUP_NAME": "TESTERGROUP1", "TEST_ID": "20", "TEST_NAME": "INTEGRATIONTEST", "TEST_VALUE": "1", "TEST_STATUS": "NOK"}
{"BATCH_TIMESTAMP": "2019-01-02T04:04:51.931+01:00", "BATCH_SOFTWARE": "1.0", "BATCH_HARDWARE": "1.0", "PLANT_LINE": "PRODLINE1", "PLANT_TESTER": "TESTER1" "PRODUCT_ID": "3", "PRODUCT_NAME": "MOTHERBOARD", "GROUP_ID": "2", "GROUP_NAME": "TESTERGROUP1", "TEST_ID": "10", "TEST_NAME": "VISUALTEST", "TEST_VALUE": "22", "TEST_STATUS": "OK"}

```

Is that possible to achieve it without any programming skills? If so, could you give me a hint on which plugins/modules should I look at?  
Thanks in advance!

---

<div class="post-metadata">

**Author:** ![Ganesh2303](https://avatars.discourse-cdn.com/v4/letter/g/57b2e6/32.png) [@Ganesh2303](https://discuss.elastic.co/u/Ganesh2303)\
**Post date:** [March 13, 2019, 12:28pm UTC](https://discuss.elastic.co/t/parsing-multiline-netsted-xml-using-logstash/172142/2 "2019-03-13T12:28:07Z")

</div>

yes you can parse this kind of log by using multline filter in input

if you're using filebeat by using to send log mean you can try this in filebeat like this,

multiline:  
pattern: ' \</BATCH\>;'  
negate: 'true'  
match: before

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 13, 2019, 1:05pm UTC](https://discuss.elastic.co/t/parsing-multiline-netsted-xml-using-logstash/172142/3 "2019-03-13T13:05:01Z")

</div>

Your XML is not valid XML. If I add a \</PRODUCT\> tag then, assuming you use a multiline codec to combine it into a single event, it can be parsed using

```
    xml { source => "message" target => "theXML" store_xml => true }
    split { field => "[theXML][PRODUCT][0][GROUP]" }
    split { field => "[theXML][PRODUCT][0][GROUP][TEST]" }

```

That will give you events that look like this:

```
    "theXML" => {
     "HARDWARE" => "1.0",
    "TIMESTAMP" => "2019-01-02T04:04:51.931+01:00",
     "SOFTWARE" => "1.0",
        "PLANT" => [
        [0] {
            "TESTER" => "TESTER1",
              "NAME" => "PARIS",
              "LINE" => "PRODLINE1"
        }
    ],
      "PRODUCT" => [
        [0] {
                "ID" => "3",
              "NAME" => "MOTHERBOARD",
             "GROUP" => {
                "TEST" => {
                    "STATUS" => "OK",
                        "ID" => "10",
                      "NAME" => "VISUALTEST",
                     "VALUE" => "22"
                },
                  "ID" => "2",
                "NAME" => "TESTGROUP2"
            },
            "STATUS" => "OK"
        }
    ]
}

```

Then you can use mutate+rename to move the fields around any way you want.

---

<div class="post-metadata">

**Author:** ![saif3r](https://avatars.discourse-cdn.com/v4/letter/s/49beb7/32.png) [@saif3r](https://discuss.elastic.co/u/saif3r)\
**Post date:** [March 13, 2019, 1:14pm UTC](https://discuss.elastic.co/t/parsing-multiline-netsted-xml-using-logstash/172142/4 "2019-03-13T13:14:40Z")

</div>

Thank you. My intention is not to use filebeat, and to rely only on Logstash pulls from the given file location.

---

<div class="post-metadata">

**Author:** ![saif3r](https://avatars.discourse-cdn.com/v4/letter/s/49beb7/32.png) [@saif3r](https://discuss.elastic.co/u/saif3r)\
**Post date:** [March 13, 2019, 1:15pm UTC](https://discuss.elastic.co/t/parsing-multiline-netsted-xml-using-logstash/172142/5 "2019-03-13T13:15:32Z")

</div>

You are correct, my apologies for the confusion. I edited my initial post and closed PRODUCT tag.  
Thanks for the proposal, I will take a look at it 🙂

---

<div class="post-metadata">

**Author:** ![Ganesh2303](https://avatars.discourse-cdn.com/v4/letter/g/57b2e6/32.png) [@Ganesh2303](https://discuss.elastic.co/u/Ganesh2303)\
**Post date:** [March 13, 2019, 1:29pm UTC](https://discuss.elastic.co/t/parsing-multiline-netsted-xml-using-logstash/172142/6 "2019-03-13T13:29:00Z")

</div>

> [@saif3r](#):
>
> My intention is not to use filebeat, and to rely only on Logstash pulls from the given file location.

You can use the multiline function in logstash also in the input section.

---

<div class="post-metadata">

**Author:** ![saif3r](https://avatars.discourse-cdn.com/v4/letter/s/49beb7/32.png) [@saif3r](https://discuss.elastic.co/u/saif3r)\
**Post date:** [March 13, 2019, 1:30pm UTC](https://discuss.elastic.co/t/parsing-multiline-netsted-xml-using-logstash/172142/7 "2019-03-13T13:30:35Z")

</div>

> [@Ganesh2303](#):
>
> You can use the multiline function in logstash also in the input section.

Will check it out, thanks!

---

<div class="post-metadata">

**Author:** ![Ganesh2303](https://avatars.discourse-cdn.com/v4/letter/g/57b2e6/32.png) [@Ganesh2303](https://discuss.elastic.co/u/Ganesh2303)\
**Post date:** [March 13, 2019, 1:32pm UTC](https://discuss.elastic.co/t/parsing-multiline-netsted-xml-using-logstash/172142/8 "2019-03-13T13:32:47Z")

</div>

codec =\> multiline {  
pattern =\> "\</BATCH\>"  
negate =\> true  
what =\> "previous"  
}

like this you can use in logstash input plugin

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 10, 2019, 1:32pm UTC](https://discuss.elastic.co/t/parsing-multiline-netsted-xml-using-logstash/172142/9 "2019-04-10T13:32:50Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.