# Split multi- nested XML in Logstash

**URL:** https://discuss.elastic.co/t/split-multi-nested-xml-in-logstash/220077
**Category:** Logstash
**Created:** [February 20, 2020, 4:41am UTC](https://discuss.elastic.co/t/split-multi-nested-xml-in-logstash/220077 "2020-02-20T04:41:15Z")
**Posts on this page:** 13
**Page:** 1

<div class="post-metadata">

### Author: ![katara](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/katara/32/60143_2.png) [@katara](https://discuss.elastic.co/u/katara)
#### Post date: [February 20, 2020, 4:41am UTC](https://discuss.elastic.co/t/split-multi-nested-xml-in-logstash/220077/1 "2020-02-20T04:41:15Z")

</div>

Hi All,

Here's a sample of my xml data

> ```
> <metric-datas>
> <metric-data>
> <metricId>120778</metricId>
> <metricName> Average Response Time (ms) </metricName>
> <metricPath> /ordering/Login.aspx </metricPath>
> <frequency>ONE_MIN</frequency>
> <metricValues>
> <metric-value>
> <startTimeInMillis>15821000</startTimeInMillis>
> <occurrences>3</occurrences>
> <current>0</current>
> <min>0</min>
> <max>47</max>
> <useRange>true</useRange>
> <count>803</count>
> <sum>1770</sum>
> <value>2</value>
> <standardDeviation>0</standardDeviation>
> </metric-value>
> </metricValues>
> </metric-data>
> </metric-datas>
> 
> ```

The above has multiple nests of XML which I want to avoid and get only the values of metrics inside.

My expected output is:

> ```
> {
> metricId: 120778
> metricName: Average Response Time (ms)
> metricPath: /ordering/Login.aspx
> frequency: ONE_MIN
> startTimeInMillis: 15821000
> occurrences: 3
> current: 0
> min: 0
> max: 47
> useRange: true
> count:803
> sum:1770
> value:2
> standardDeviation: 0
> }
> 
> ```

as seperate row fields.

Here's my current conf file, which is faulty, however, I hope I can get it corrected with your help.

> ```
> input {
> http_poller {
> urls => {
> url => "https://appd.com/controller/virtual%20Response%20Time%20%28ms%29&time-range-type=BEFORE_NOW&duration-in-mins=5"
> }
> truststore => "path/to/cacerts.jks"
> truststore_password => " *****"
> request_timeout => 60
> user => " *****"
> password => " ****"
> metadata_target => "http_poller_metadata"
> schedule => { cron => "* * * * * UTC"}
> }
> }
> filter
> {
> xml {
> source => "[metric-datas]"
> store_xml => "false"
> }
> split {
> field => "[metric-datas]"
> }
> 
> }
> output {
> elasticsearch {
> hosts => ["10.1.533.209:9200"]
> index => "appdmetric"
> }
> stdout { codec => rubydebug }
> }
> 
> ```

Thank you!

Katara

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [February 20, 2020, 3:23pm UTC](https://discuss.elastic.co/t/split-multi-nested-xml-in-logstash/220077/2 "2020-02-20T15:23:32Z")

</div>

If store\_xml is false, and there are no xpath expressions, then all the xml filter does is validate the XML (and it is rather liberal in doing that). Also, if you change store\_xml to true then you will not have a metric-datas field, the field inside target will be [metric-data].

---

<div class="post-metadata">

### Author: ![katara](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/katara/32/60143_2.png) [@katara](https://discuss.elastic.co/u/katara)
#### Post date: [February 21, 2020, 10:51am UTC](https://discuss.elastic.co/t/split-multi-nested-xml-in-logstash/220077/3 "2020-02-21T10:51:32Z")

</div>

Thank you @Badger, for helping me out,

I changed a few as per your suggestion,

> ```
> filter
> {
> xml {
> source => "[metric-data]"
> target => "xmldata"
> store_xml => "true"
> }
> split {
> field => "[metric-data]"
> }
>      
> }
> 
> ```

However, my data is sill unsplit.

Anything im missing?  
There has to be a data split in  
`<metric- data>`  
and also in  
`<metricValues> <metric-value>`

Katara.

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [February 21, 2020, 3:23pm UTC](https://discuss.elastic.co/t/split-multi-nested-xml-in-logstash/220077/4 "2020-02-21T15:23:33Z")

</div>

What does the rubydebug output on stdout look like?

---

<div class="post-metadata">

### Author: ![katara](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/katara/32/60143_2.png) [@katara](https://discuss.elastic.co/u/katara)
#### Post date: [February 21, 2020, 4:21pm UTC](https://discuss.elastic.co/t/split-multi-nested-xml-in-logstash/220077/5 "2020-02-21T16:21:10Z")

</div>

@Badger, It shows the entire xml data under message.

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [February 21, 2020, 5:20pm UTC](https://discuss.elastic.co/t/split-multi-nested-xml-in-logstash/220077/6 "2020-02-21T17:20:49Z")

</div>

If your XML is in a field called message then why are you telling the xml filter to look at a field called metric-data?

---

<div class="post-metadata">

### Author: ![katara](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/katara/32/60143_2.png) [@katara](https://discuss.elastic.co/u/katara)
#### Post date: [February 24, 2020, 4:43am UTC](https://discuss.elastic.co/t/split-multi-nested-xml-in-logstash/220077/7 "2020-02-24T04:43:23Z")

</div>

@Badger, Okay ,  
I see what I'm missing ,  
I changed my filter to the below,

> ```
> filter
> {
> xml {
> source => "message"
> target => "xmldata"
> store_xml => "true"
> }
> split {
> field => "message"
> }
> 
> }
> 
> ```

and I get this,  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/6/7673eeba9ed6e9605b2fdd66068f2a8f9aa1de2a.png)

How do I eliminate the tags and make them a valid field name where necessary?  
Kindly help me out!

Katara

---

<div class="post-metadata">

### Author: ![katara](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/katara/32/60143_2.png) [@katara](https://discuss.elastic.co/u/katara)
#### Post date: [February 26, 2020, 12:42pm UTC](https://discuss.elastic.co/t/split-multi-nested-xml-in-logstash/220077/9 "2020-02-26T12:42:11Z")

</div>

I did some research and tried the below with a multiline:

> ```
> input {
> http_poller {
> urls => {
> url => "https://appd.com/controller/virtual%20Response%20Time%20%28ms%29&time-range-type=BEFORE_NOW&duration-in-mins=5"
> }
> truststore => "path/to/cacerts.jks"
> truststore_password => " *****"
> request_timeout => 60
> user => " *****"
> password => " ****"
> metadata_target => "http_poller_metadata"
> schedule => { cron => "* * * * * UTC"}
> codec => multiline {
> pattern => "<metric-datas>"
> negate => "true"
> what => "previous"
> }
> }
> }
> filter
> {
> xml {
> source => "message"
> target => "xmldata"
> store_xml => "true"
> xpath => [
> "/metric-datas/metric-data/metricID/text()", "metricID",
> "/metric-datas/metric-data/metricName/text()", "metricName",
> "/metric-datas/metric-data/metricPath/text()", "metricPath",
> "/metric-datas/metric-data/frequency/text()", "Frequency",
> "/metric-datas/metric-data/metricValues/metric-value/startTimeInMillis/text()", "starttime",
> "/metric-datas/metric-data/metricValues/metric-value/occurences/text()", "occurences",
> "/metric-datas/metric-data/metricValues/metric-value/current/text()", "current",
> "/metric-datas/metric-data/metricValues/metric-value/min/text()", "min",
> "/metric-datas/metric-data/metricValues/metric-value/max/text()", "max",
> "/metric-datas/metric-data/metricValues/metric-value/useRange/text()", "UseRange",
> "/metric-datas/metric-data/metricValues/metric-value/count/text()", "count",
> "/metric-datas/metric-data/metricValues/metric-value/sum/text()", "sum",
> "/metric-datas/metric-data/metricValues/metric-value/value/text()", "value",
> "/metric-datas/metric-data/metricValues/metric-value/standardDeviation/text()", "SatndardDeviation",
> ]
> }
> 
> }
> 
> ```

which is giving me an error:

`

> [2020-02-26T07:38:23,938][ERROR][logstash.agent] Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:appd, :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of #, ", ', -, [, { at line 42, column 9 (byte 2006) after filter\n{\nxml {\n source =\> "message"\n\t\ttarget =\> "xmldata"\n store\_xml =\> "true"\nxpath =\> [\n "/metric-datas/metric-data/metricID/text()", "metricID",\n "/metric-datas/metric-data/metricName/text()", "metricName",\n\t "/metric-datas/metric-data/metricPath/text()", "metricPath",\n\t "/metric-datas/metric-data/frequency/text()", "Frequency",\n\t "/metric-datas/metric-data/metricValues/metric-value/startTimeInMillis/text()", "starttime",\n\t "/metric-datas/metric-data/metricValues/metric-value/occurences/text()", "occurences",\n\t "/metric-datas/metric-data/metricValues/metric-value/current/text()", "current",\n\t "/metric-datas/metric-data/metricValues/metric-value/min/text()", "min",\n\t "/metric-datas/metric-data/metricValues/metric-value/max/text()", "max",\n\t "/metric-datas/metric-data/metricValues/metric-value/useRange/text()", "UseRange",\n\t "/metric-datas/metric-data/metricValues/metric-value/count/text()", "count",\n\t "/metric-datas/metric-data/metricValues/metric-value/sum/text()", "sum",\n\t "/metric-datas/metric-data/metricValues/metric-value/value/text()", "value",\n\t "/metric-datas/metric-data/metricValues/metric-value/standardDeviation/text()", "SatndardDeviation",\n ", :backtrace=\>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in `compile\_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2577:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:151:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:24:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline\_action/create.rb:36:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:325:in `block in converge\_state'"]}

`

Not sure if im doing this right.

@Badger , pelase help me correct this

Katara

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [February 26, 2020, 3:06pm UTC](https://discuss.elastic.co/t/split-multi-nested-xml-in-logstash/220077/10 "2020-02-26T15:06:42Z")

</div>

> [@katara](#):
>
> "SatndardDeviation",

You cannot have a comma after the last entry in an array.

---

<div class="post-metadata">

### Author: ![katara](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/katara/32/60143_2.png) [@katara](https://discuss.elastic.co/u/katara)
#### Post date: [February 27, 2020, 4:22am UTC](https://discuss.elastic.co/t/split-multi-nested-xml-in-logstash/220077/11 "2020-02-27T04:22:31Z")

</div>

@badger,  
I must have missed it. I corrected it and I still do not get the separate field.  
Am I using the conditions right?  
Please help me out.

Katara

---

<div class="post-metadata">

### Author: ![katara](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/katara/32/60143_2.png) [@katara](https://discuss.elastic.co/u/katara)
#### Post date: [February 27, 2020, 5:07am UTC](https://discuss.elastic.co/t/split-multi-nested-xml-in-logstash/220077/12 "2020-02-27T05:07:01Z")

</div>

If it helps anyone,  
This is what worked for me, setting ` force_array => false` is another change I made.

> ```
> filter
> {
> xml {
> source => "message"
> target => "xmldata"
> store_xml => "true"
> force_array => false
> xpath => [
> "/metric-datas/metric-data/metricID/text()", "metricID",
> "/metric-datas/metric-data/metricName/text()", "metricName",
> "/metric-datas/metric-data/metricPath/text()", "metricPath",
> "/metric-datas/metric-data/frequency/text()", "Frequency",
> "/metric-datas/metric-data/metricValues/metric-value/startTimeInMillis/text()", "starttime",
> "/metric-datas/metric-data/metricValues/metric-value/occurences/text()", "occurences",
> "/metric-datas/metric-data/metricValues/metric-value/current/text()", "current",
> "/metric-datas/metric-data/metricValues/metric-value/min/text()", "min",
> "/metric-datas/metric-data/metricValues/metric-value/max/text()", "max",
> "/metric-datas/metric-data/metricValues/metric-value/useRange/text()", "UseRange",
> "/metric-datas/metric-data/metricValues/metric-value/count/text()", "count",
> "/metric-datas/metric-data/metricValues/metric-value/sum/text()", "sum",
> "/metric-datas/metric-data/metricValues/metric-value/value/text()", "value",
> "/metric-datas/metric-data/metricValues/metric-value/standardDeviation/text()", "StandardDeviation"
> ]
> }
> mutate {
> remove_field => ["message","[http_poller_metadata][request][url]",
> "[http_poller_metadata][response_headers][x-xss-protection]",
> "[http_poller_metadata][response_headers][set-cookie]",
> "[http_poller_metadata][response_headers][x-content-type-options]",
> "[http_poller_metadata][response_headers][transfer-encoding]",
> "[http_poller_metadata][response_headers][content-type]",
> "[http_poller_metadata][response_headers][x-frame-options]",
> "[http_poller_metadata][response_message]",
> "[http_poller_metadata][request.method]",
> "[http_poller_metadata][code]",
> "[http_poller_metadata][times_retried]",
> "[http_poller_metadata][name]",
> "[http_poller_metadata][runtime_seconds]"]
> }
> }
> 
> ```

Hope it helps!

Thanks 🙂  
Katara

---

<div class="post-metadata">

### Author: ![katara](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/katara/32/60143_2.png) [@katara](https://discuss.elastic.co/u/katara)
#### Post date: [February 27, 2020, 6:40am UTC](https://discuss.elastic.co/t/split-multi-nested-xml-in-logstash/220077/13 "2020-02-27T06:40:06Z")

</div>

Even better,

> ```
> filter
> {
> xml {
> source => "message"
> store_xml => "true"
> target => "xmldata"
> force_array => false
> }
> 
> ```

simply this works with the input ive given!

Cheers! 🙂

Katara.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 26, 2020, 6:40am UTC](https://discuss.elastic.co/t/split-multi-nested-xml-in-logstash/220077/14 "2020-03-26T06:40:08Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.