# Nested XML with Logstash- Recommendation needed

**URL:** <https://discuss.elastic.co/t/nested-xml-with-logstash-recommendation-needed/106333>\
**Category:** Logstash\
**Created:** [November 3, 2017, 3:26pm UTC](https://discuss.elastic.co/t/nested-xml-with-logstash-recommendation-needed/106333 "2017-11-03T15:26:41Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![moriol](https://avatars.discourse-cdn.com/v4/letter/m/439d5e/32.png) [@moriol](https://discuss.elastic.co/u/moriol)\
**Post date:** [November 3, 2017, 3:26pm UTC](https://discuss.elastic.co/t/nested-xml-with-logstash-recommendation-needed/106333/1 "2017-11-03T15:26:41Z")

</div>

Hello,

I am looking for some recommendations, where I have an XML file which is nested and needs to be an input in Logstash.  
My xml has the following structure:  
\< subsystem\>  
\< component\>  
\< Parameter\>  
\< /Parameter\>  
\< Parameter\>  
\< /Parameter\>  
\< /component\>  
\< component2\>  
\< Parameter\>  
\< /Parameter\>  
\< Parameter\>  
\< /Parameter\>  
\< /component2\>  
\< /subsystem\>

\< subsystem2\>  
\< component\>  
\< Parameter\>  
\< /Parameter\>  
\< Parameter\>  
\< /Parameter\>  
\< /component\>  
\< component2\>  
\< Parameter\>  
\< /Parameter\>  
\< Parameter\>  
\< /Parameter\>  
\< /component2\>  
\< /subsystem2\>

Basically what I am trying to achieve is to get all parameters into ElasticSearch and add additional fields, with what component and subsystem they belong to. I have about 10 different subsystems in the file, each has different components, where each have different parameters. Whats the best way to approach this?  
So far I have a Logstash file, which just lists all the fields, but parameters are not linked to their subsystem or components.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 3, 2017, 4:55pm UTC](https://discuss.elastic.co/t/nested-xml-with-logstash-recommendation-needed/106333/2 "2017-11-03T16:55:05Z")

</div>

It is not clear what you want to achieve, so let me say what I would do with data like that. The first thing to do is use a multiline input codec to join all the parts of a \<subsystem together\>. That's going to look something like this if subsystems are separated by a blank line.

```
input {
  stdin {
    codec => multiline {
      pattern => "^$"
      negate => true
      what => previous
    }
  }
}
output { stdout { codec => rubydebug } }
```

The message will have newlines embedded in it, so they have to be mutated away. Note that there is no fancy quoting scheme to get a newline into a string, you just put the open and close quotes on different lines.

```
mutate {
    gsub => [ "message", "
", "" ]
  }
```

Then we need to parse the XML, which looks like this

```
xml {
    source => "message"
    target => "theXML"
    suppress_empty => false
    force_array => false
  }
```

If that does not do what you want, you need to show what you are doing, with input data, and what you do not like about the output.

---

<div class="post-metadata">

**Author:** ![moriol](https://avatars.discourse-cdn.com/v4/letter/m/439d5e/32.png) [@moriol](https://discuss.elastic.co/u/moriol)\
**Post date:** [November 30, 2017, 2:15pm UTC](https://discuss.elastic.co/t/nested-xml-with-logstash-recommendation-needed/106333/3 "2017-11-30T14:15:37Z")

</div>

Basically what I have is an xml file, which is nested and has the following values:  
-a number of subsystems  
-inside subsystem, a number of components  
-inside components a number of parameters  
This repeats, so i configured something like this in input in Logstash  
input {  
file {  
path =\> "/files/pipo.xml"  
start\_position =\> "beginning"  
sincedb\_path =\> "/dev/null"  
codec =\> multiline {  
pattern =\> "\<(subsystem|component|parameters)"  
negate =\> true  
what =\> "previous"  
}  
}  
}

This works well for me, except it divides all of these results into a separate json outputs, where i have a separate json for subsystem, components and then parameters- its too split up.  
I want to achieve a single json per subsystem, where inside i have all the values for components and parameters.  
I want to have it represented that way as, components and parameters refer to a specific subsystem and i need to group data by this.  
Any idea how to achieve this?

---

<div class="post-metadata">

**Author:** ![moriol](https://avatars.discourse-cdn.com/v4/letter/m/439d5e/32.png) [@moriol](https://discuss.elastic.co/u/moriol)\
**Post date:** [November 30, 2017, 2:38pm UTC](https://discuss.elastic.co/t/nested-xml-with-logstash-recommendation-needed/106333/4 "2017-11-30T14:38:34Z")

</div>

What would be even better is to have a json for each parameter, which would have the component that it belongs to, as well as subsystem to which it belongs to.  
for example  
{parameter= test1243  
component= test2  
subsystem= pipo  
}  
{parameter= 19172  
component=test2  
subsystem=pipo  
}

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 28, 2017, 2:39pm UTC](https://discuss.elastic.co/t/nested-xml-with-logstash-recommendation-needed/106333/5 "2017-12-28T14:39:16Z")

</div>

Closed on December 28, 2017, 2:39pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.