# Parsing complicated XML - extracting nested fields and splitting

**URL:** <https://discuss.elastic.co/t/parsing-complicated-xml-extracting-nested-fields-and-splitting/167863>\
**Category:** Logstash\
**Created:** [February 11, 2019, 1:18pm UTC](https://discuss.elastic.co/t/parsing-complicated-xml-extracting-nested-fields-and-splitting/167863 "2019-02-11T13:18:30Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Skwilly](https://avatars.discourse-cdn.com/v4/letter/s/5f8ce5/32.png) [@Skwilly](https://discuss.elastic.co/u/Skwilly)\
**Post date:** [February 11, 2019, 1:18pm UTC](https://discuss.elastic.co/t/parsing-complicated-xml-extracting-nested-fields-and-splitting/167863/1 "2019-02-11T13:18:31Z")

</div>

Hello,

I have the following multiline XML:

```
<SharedProject id="1">
 <m__modelerProxy id="1">
		<m__building id="140">
     <m __name>z2</m__ name>
 <m__common class="SharedBuilding" id="141">
        <m __name>z2</m__ name>
        <m__modelerProxy class="be.lucid.PEB.model.geometry.Zone" reference="140" />
        <m__energeticProxy class="Building" id="142">
           <attributes id="143">
           </attributes>
           <labels id="144" />
           <name>z2</name>
           <pebIdentifer>2</pebIdentifer>
           <pebUniqueIdentifer>2</pebUniqueIdentifer>
           <proxy class="SharedBuilding" reference="141" />
           <plot id="145">
           <protectedVolumes id="627">
		<ProtectedVolume id="628">
			<envelope id="631">
				<paroiRoles id="633">
					<ParoiRole id="634">
                                              <attributes id="635" />
                                              <labels id="636" />
                                              <name>236</name>
                                              <pebIdentifer>237</pebIdentifer>
                                              <position>SIDE</position>
                                              <paroi id="637">
					</ParoiRole>
				        <ParoiRole id="7395">
                                              <attributes id="7396" />
                                              <labels id="7397" />
                                              <name>237</name>
                                              <pebIdentifer>238</pebIdentifer>
                                              <position>SIDE</position>
                                             <paroi id="797">
				        </ParoiRole>
                                        ...
                                </paroiRoles>

```

I want to store only what's inside "ProtectedVolume" field and then split based on the "ParoiRole" field (I want separate events for this field).

I tried running the following input and filter configurations

```
input {

file {
path => "pathtoxml/file.xml"
start_position => "beginning"
sincedb_path => "NUL"
codec => multiline { pattern => "^Spalanzani" negate => true what => "previous" auto_flush_interval => 2 max_lines=>30000}
  }
}

```

and

```
filter {
xml { source => "message" target => "theXML" store_xml => true force_array => false }

split { field => "[theXML][m__modelerProxy][m__building][m__common][m__energeticProxy][protectedVolumes][ProtectedVolume][envelope][paroiRoles][ParoiRole]" remove_field => "message"}

mutate
{
     remove_field => ["message"]
}

}

```

However, I do not know how to remove all the fields before the `<ProtectedVolume>` and get the results I want.

Thanks in advance for your help,

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 11, 2019, 1:43pm UTC](https://discuss.elastic.co/t/parsing-complicated-xml-extracting-nested-fields-and-splitting/167863/2 "2019-02-11T13:43:40Z")

</div>

Many of your XML elements are not terminated. I am guessing most of the should be of the \<element attribute="a" /\> form. If not then the path inside theXML should include all the unclosed elements.

```
    xml { source => "message" target => "[theXML]" store_xml => true force_array => false }
    mutate { rename => { "[theXML][protectedVolumes][ProtectedVolume]" => "ProtectedVolume" } }
    mutate { remove_field => ["message", "theXML"] }
    split { field => "[ProtectedVolume][paroiRoles][ParoiRole]" }
```

---

<div class="post-metadata">

**Author:** ![Skwilly](https://avatars.discourse-cdn.com/v4/letter/s/5f8ce5/32.png) [@Skwilly](https://discuss.elastic.co/u/Skwilly)\
**Post date:** [February 11, 2019, 4:41pm UTC](https://discuss.elastic.co/t/parsing-complicated-xml-extracting-nested-fields-and-splitting/167863/3 "2019-02-11T16:41:18Z")

</div>

Thank you very much, this is indeed what I was looking for.

For information, each tags have closing tags in my XML file but I am sharing a simplified version because the real one has 30k lines and messier.

I have noticed an issue with my XML and its more complicated that I thought.  
There is a nested Pattern that I would like to extract but I dont know how.

Thank to you, I have successfully splitted my XML with  
`split { field => "[ProtectedVolume][envelope][paroiRoles][ParoiRole]" }`

However, this tag patterns will **sometimes** be nested further, see example below:

Inside `<ParoiRole>`, we have a `<volume>` tag then another `<paroiRoles><ParoiRole>` pattern which is repeated. In this case, I would like to store it in a separate event again.

```auto
<SharedProject id="1">
 <m__modelerProxy id="1">
		<m__building id="140">
     <m __name>z2</m__ name>
 <m__common class="SharedBuilding" id="141">
        <m __name>z2</m__ name>
        <m__modelerProxy class="be.lucid.PEB.model.geometry.Zone" reference="140" />
        <m__energeticProxy class="Building" id="142">
           <attributes id="143">
           </attributes>
           <labels id="144" />
           <name>z2</name>
           <pebIdentifer>2</pebIdentifer>
           <pebUniqueIdentifer>2</pebUniqueIdentifer>
           <proxy class="SharedBuilding" reference="141" />
           <plot id="145">
           <protectedVolumes id="627">
		<ProtectedVolume id="628">
			<envelope id="631">
				<paroiRoles id="633">
					<ParoiRole id="634">
                                              <attributes id="635" />
                                              <labels id="636" />
                                              <name>236</name>
                                              <pebIdentifer>237</pebIdentifer>
                                              <position>SIDE</position>
                                              <paroi id="637">
					</ParoiRole>
				        <ParoiRole id="7395">
                                              <attributes id="7396" />
                                              <labels id="7397" />
                                              <name>237</name>
                                              <pebIdentifer>238</pebIdentifer>
                                              <position>SIDE</position>
                                             <paroi id="797">
				        </ParoiRole>
                                        <ParoiRole>
                                           <attributes id="7398" />
                                           <labels id="7399" />
                                           <volume>
                                             <paroiRoles>
                                                <ParoiRole>
                                                   ...
                                                  
                                        ...
                                </paroiRoles>

```

In my case, I actually have 17 ParoiRole nested in the first one that end up in a single document when indexed. I am not sure which approach is best, the end-goal would be to generate statistics on fields containted in ParoiRole. The queries are messed up right now because of this missed nested pattern.

Is my request achievable through logstash ? Or should I ask for cleaner data or transform it myself with another tool ?

I hope my post was clear enough, I must admit to be a newbie working with a confusing XML..

Thanks and regards,

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 11, 2019, 4:58pm UTC](https://discuss.elastic.co/t/parsing-complicated-xml-extracting-nested-fields-and-splitting/167863/4 "2019-02-11T16:58:14Z")

</div>

You can do a second split if the nested field exists.

```
    split { field => "[ProtectedVolume][envelope][paroiRoles][ParoiRole]" }
    if [ProtectedVolume][envelope][paroiRoles][ParoiRole][volume][paroiRoles] {
        split { field => "[ProtectedVolume][envelope][paroiRoles][ParoiRole][volume][paroiRoles]" }
    }
```

---

<div class="post-metadata">

**Author:** ![Skwilly](https://avatars.discourse-cdn.com/v4/letter/s/5f8ce5/32.png) [@Skwilly](https://discuss.elastic.co/u/Skwilly)\
**Post date:** [February 12, 2019, 10:00am UTC](https://discuss.elastic.co/t/parsing-complicated-xml-extracting-nested-fields-and-splitting/167863/5 "2019-02-12T10:00:15Z")

</div>

Thanks, this gives me exactly what I needed.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 12, 2019, 10:00am UTC](https://discuss.elastic.co/t/parsing-complicated-xml-extracting-nested-fields-and-splitting/167863/6 "2019-03-12T10:00:15Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.