# Turning one .xml into multiple events where not all values are always filled

**URL:** <https://discuss.elastic.co/t/turning-one-xml-into-multiple-events-where-not-all-values-are-always-filled/176384>\
**Category:** Logstash\
**Created:** [April 11, 2019, 10:50am UTC](https://discuss.elastic.co/t/turning-one-xml-into-multiple-events-where-not-all-values-are-always-filled/176384 "2019-04-11T10:50:50Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![saif3r](https://avatars.discourse-cdn.com/v4/letter/s/49beb7/32.png) [@saif3r](https://discuss.elastic.co/u/saif3r)\
**Post date:** [April 11, 2019, 10:50am UTC](https://discuss.elastic.co/t/turning-one-xml-into-multiple-events-where-not-all-values-are-always-filled/176384/1 "2019-04-11T10:50:51Z")

</div>

Hello All,

I'm learning Logstash basics and I'm trying to find a way of converting an .xml file into multiple events (separate json files) which will be sent out to elasticsearch. The thing is, specific elements of xml might vary in terms of content. One event might have test info, the other one not, etc. More details below. Each .xml consists of two lines, one xml metadata and one content line. I'm interested only in content lines. Here's source file example:

```
<?xml version="1.0" encoding="UTF-8"?>
<BATCH TIMESTAMP="2019-01-02T04:04:51.931+01:00" SOFTWARE="1.0"><PLANT NAME="PARIS" LINE="PRODLINE1"/><PRODUCT ID="3" NAME="MOTHERBOARD"><GROUP ID="1" NAME="TESTGROUP1"><TEST ID="10" NAME="VOLTAGETEST" VALUE="2.34523" STATUS="OK"/><TEST ID="20" NAME="INTEGRATION" VALUE="1.00000" STATUS="NOK"/></GROUP><GROUP ID="2" NAME="CHANGEOVER">DONE</GROUP><GROUP ID="3" NAME="NOTIFICATION"><EXTRA TEXT="OK"/></GROUP><GROUP ID="4" NAME="VOLTAGE_NOTIF"><TEST ID="10" NAME="VOLTAGETEST" VALUE="4.00001" STATUS="NOK"/><EXTRA TEXT="OK"/></GROUP></PRODUCT></BATCH>

```

Formatted for better readability:

```
<?xml version="1.0" encoding="UTF-8"?>
<BATCH TIMESTAMP="2019-01-02T04:04:51.931+01:00" SOFTWARE="1.0">
<PLANT NAME="PARIS" LINE="PRODLINE1"/>
<PRODUCT ID="3" NAME="MOTHERBOARD">
	<GROUP ID="1" NAME="TESTGROUP1">
		<TEST ID="10" NAME="VOLTAGETEST" VALUE="2.34523" STATUS="OK"/>
		<TEST ID="20" NAME="INTEGRATION" VALUE="1.00000" STATUS="NOK"/>
	</GROUP>
	<GROUP ID="2" NAME="CHANGEOVER">DONE</GROUP>
	<GROUP ID="3" NAME="NOTIFICATION">
		<EXTRA TEXT="OK"/>
	</GROUP>
	<GROUP ID="4" NAME="VOLTAGE_NOTIF">
		<TEST ID="10" NAME="VOLTAGETEST" VALUE="4.00001" STATUS="NOK"/>
		<EXTRA TEXT="OK"/>
	</GROUP>
</PRODUCT>
</BATCH>

```

And here's desired results - 5 separate events:

```
{"BATCH_TIMESTAMP": "2019-01-02T04:04:51.931+01:00", "BATCH_SOFTWARE": "1.0", "PLANT_NAME": "PARIS", "PLANT_LINE": "PRODLINE1", "PRODUCT_ID": "3", "PRODUCT_NAME": "MOTHERBOARD", "GROUP_ID": "1", "GROUP_NAME": "TESTERGROUP1", "GROUP_VALUE": "", "TEST_ID": "10", "TEST_NAME": "VOLTAGETEST", "TEST_VALUE": "2.34523", "TEST_STATUS": "OK", "EXTRA_TEXT": ""}
{"BATCH_TIMESTAMP": "2019-01-02T04:04:51.931+01:00", "BATCH_SOFTWARE": "1.0", "PLANT_NAME": "PARIS", "PLANT_LINE": "PRODLINE1", "PRODUCT_ID": "3", "PRODUCT_NAME": "MOTHERBOARD", "GROUP_ID": "1", "GROUP_NAME": "TESTERGROUP1", "GROUP_VALUE": "", "TEST_ID": "20", "TEST_NAME": "INTEGRATION", "TEST_VALUE": "1.00000", "TEST_STATUS": "NOK", "EXTRA_TEXT": ""}
{"BATCH_TIMESTAMP": "2019-01-02T04:04:51.931+01:00", "BATCH_SOFTWARE": "1.0", "PLANT_NAME": "PARIS", "PLANT_LINE": "PRODLINE1", "PRODUCT_ID": "3", "PRODUCT_NAME": "MOTHERBOARD", "GROUP_ID": "2", "GROUP_NAME": "CHANGEOVER", "GROUP_VALUE": "DONE", "TEST_ID": "", "TEST_NAME": "", "TEST_VALUE": "", "TEST_STATUS": "", "EXTRA_TEXT": ""}
{"BATCH_TIMESTAMP": "2019-01-02T04:04:51.931+01:00", "BATCH_SOFTWARE": "1.0", "PLANT_NAME": "PARIS", "PLANT_LINE": "PRODLINE1", "PRODUCT_ID": "3", "PRODUCT_NAME": "MOTHERBOARD", "GROUP_ID": "3", "GROUP_NAME": "NOTIFICATION", "GROUP_VALUE": "", "TEST_ID": "", "TEST_NAME": "", "TEST_VALUE": "", "TEST_STATUS": "", "EXTRA_TEXT": "OK"}
{"BATCH_TIMESTAMP": "2019-01-02T04:04:51.931+01:00", "BATCH_SOFTWARE": "1.0", "PLANT_NAME": "PARIS", "PLANT_LINE": "PRODLINE1", "PRODUCT_ID": "3", "PRODUCT_NAME": "MOTHERBOARD", "GROUP_ID": "4", "GROUP_NAME": "VOLTAGE_NOTIF", "GROUP_VALUE": "", "TEST_ID": "10", "TEST_NAME": "VOLTAGETEST", "TEST_VALUE": "4.00001", "TEST_STATUS": "NOK", "EXTRA_TEXT": "OK"}

```

Formatted and explained below:

- Event 1 (Group 1, Test 10, No Extra Text):

> {  
> "BATCH\_TIMESTAMP":"2019-01-02T04:04:51.931+01:00",  
> "BATCH\_SOFTWARE":"1.0",  
> "PLANT\_NAME":"PARIS",  
> "PLANT\_LINE":"PRODLINE1",  
> "PRODUCT\_ID":"3",  
> "PRODUCT\_NAME":"MOTHERBOARD",  
> "GROUP\_ID":"1",  
> "GROUP\_NAME":"TESTERGROUP1",  
> "GROUP\_VALUE":"",  
> "TEST\_ID":"10",  
> "TEST\_NAME":"VOLTAGETEST",  
> "TEST\_VALUE":"2.34523",  
> "TEST\_STATUS":"OK",  
> "EXTRA\_TEXT":""  
> }

- Event 2 (Group 1, Test 20, No Extra Text):

> {  
> "BATCH\_TIMESTAMP":"2019-01-02T04:04:51.931+01:00",  
> "BATCH\_SOFTWARE":"1.0",  
> "PLANT\_NAME":"PARIS",  
> "PLANT\_LINE":"PRODLINE1",  
> "PRODUCT\_ID":"3",  
> "PRODUCT\_NAME":"MOTHERBOARD",  
> "GROUP\_ID":"1",  
> "GROUP\_NAME":"TESTERGROUP1",  
> "GROUP\_VALUE":"",  
> "TEST\_ID":"20",  
> "TEST\_NAME":"INTEGRATION",  
> "TEST\_VALUE":"1.00000",  
> "TEST\_STATUS":"NOK",  
> "EXTRA\_TEXT":""  
> }

- Event 3 (Group 2, No Test, No Extra Text):

> {  
> "BATCH\_TIMESTAMP":"2019-01-02T04:04:51.931+01:00",  
> "BATCH\_SOFTWARE":"1.0",  
> "PLANT\_NAME":"PARIS",  
> "PLANT\_LINE":"PRODLINE1",  
> "PRODUCT\_ID":"3",  
> "PRODUCT\_NAME":"MOTHERBOARD",  
> "GROUP\_ID":"2",  
> "GROUP\_NAME":"CHANGEOVER",  
> "GROUP\_VALUE":"DONE",  
> "TEST\_ID":"",  
> "TEST\_NAME":"",  
> "TEST\_VALUE":"",  
> "TEST\_STATUS":"",  
> "EXTRA\_TEXT":""  
> }

- Event 4 (Group 3, No Text, Extra Text Present):

> {  
> "BATCH\_TIMESTAMP":"2019-01-02T04:04:51.931+01:00",  
> "BATCH\_SOFTWARE":"1.0",  
> "PLANT\_NAME":"PARIS",  
> "PLANT\_LINE":"PRODLINE1",  
> "PRODUCT\_ID":"3",  
> "PRODUCT\_NAME":"MOTHERBOARD",  
> "GROUP\_ID":"3",  
> "GROUP\_NAME":"NOTIFICATION",  
> "GROUP\_VALUE":"",  
> "TEST\_ID":"",  
> "TEST\_NAME":"",  
> "TEST\_VALUE":"",  
> "TEST\_STATUS":"",  
> "EXTRA\_TEXT":"OK"  
> }

- Event 5 (Group 4, Test 10, Extra Text Present):

> {  
> "BATCH\_TIMESTAMP":"2019-01-02T04:04:51.931+01:00",  
> "BATCH\_SOFTWARE":"1.0",  
> "PLANT\_NAME":"PARIS",  
> "PLANT\_LINE":"PRODLINE1",  
> "PRODUCT\_ID":"3",  
> "PRODUCT\_NAME":"MOTHERBOARD",  
> "GROUP\_ID":"4",  
> "GROUP\_NAME":"VOLTAGE\_NOTIF",  
> "GROUP\_VALUE":"",  
> "TEST\_ID":"10",  
> "TEST\_NAME":"VOLTAGETEST",  
> "TEST\_VALUE":"4.00001",  
> "TEST\_STATUS":"NOK",  
> "EXTRA\_TEXT":"OK"  
> }

I'm trying to figure it out using both xml and split filters, but no success as of yet. I'd appreciate the suggestions especially on how to handle different structures of event if they are not present in the source like in Event 3 (tests, extra text).  
Thanks in advance!

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 11, 2019, 12:55pm UTC](https://discuss.elastic.co/t/turning-one-xml-into-multiple-events-where-not-all-values-are-always-filled/176384/2 "2019-04-11T12:55:29Z")

</div>

I suggest you start with

```
    xml { source => "message" target => "theXML" }
    split { field => "[theXML][PRODUCT][0][GROUP]" }
    split { field => "[theXML][PRODUCT][0][GROUP][TEST]" }

```

and then add a whole bunch of mutate+rename filters.

---

<div class="post-metadata">

**Author:** ![saif3r](https://avatars.discourse-cdn.com/v4/letter/s/49beb7/32.png) [@saif3r](https://discuss.elastic.co/u/saif3r)\
**Post date:** [April 18, 2019, 11:57am UTC](https://discuss.elastic.co/t/turning-one-xml-into-multiple-events-where-not-all-values-are-always-filled/176384/3 "2019-04-18T11:57:58Z")

</div>

Hi @Badger,  
Thanks. Could you tell me what's the purpose of [0] within a split filter?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 18, 2019, 12:54pm UTC](https://discuss.elastic.co/t/turning-one-xml-into-multiple-events-where-not-all-values-are-always-filled/176384/4 "2019-04-18T12:54:12Z")

</div>

> [@saif3r](#):
>
> Thanks. Could you tell me what's the purpose of [0] within a split filter?

The xml filter has a [force\_array](https://www.elastic.co/guide/en/logstash/current/plugins-filters-xml.html#plugins-filters-xml-force_array) option. If you do not set that to false, then [theXML][PRODUCT] will be an array, even though there is only one such element. The [0] references the first (and only) entry in the array.

---

<div class="post-metadata">

**Author:** ![saif3r](https://avatars.discourse-cdn.com/v4/letter/s/49beb7/32.png) [@saif3r](https://discuss.elastic.co/u/saif3r)\
**Post date:** [April 18, 2019, 1:09pm UTC](https://discuss.elastic.co/t/turning-one-xml-into-multiple-events-where-not-all-values-are-always-filled/176384/5 "2019-04-18T13:09:06Z")

</div>

I see, thank you. I noticed that Product is an array, which is slightly inconvenient, since the easiest way to handle this data set is to have it flatten. With this xml as a example, how can i make sure that each event is stored as a separate event, but with no arrays? Instead, the index should have each value stored as a separate field.

Here's the part of current elastic index:

```
theXML.PLANT {
  "LINE": "PRODLINE1",
  "NAME": "PARIS"
}
     	? theXML.PRODUCT {
  "GROUP": {
    "TEST": {
      "VALUE": "2.34523",
      "NAME": "VOLTAGETEST",
      "STATUS": "OK",
      "ID": "10"
    },
    "NAME": "TESTGROUP1",
    "ID": "1"
  },
  "NAME": "MOTHERBOARD",
  "ID": "3"
}

```

![2019-04-18%2015_11_33-Discover%20-%20Kibana](https://us1.discourse-cdn.com/elastic/original/3X/0/c/0cb04e4d6d9b646e7938dfc162199f95db614a17.png)

How could I achieve something like this?

> {  
> theXML\_PLANT\_NAME: "PARIS",  
> theXML\_PLANT\_LINE: "PRODLINE1"  
> theXML\_PRODUCT\_ID: "3",  
> theXML\_PRODUCT\_NAME: "MOTHERBOARD",  
> theXML\_GROUP\_ID: "1",  
> theXML\_GROUP\_NAME: "TESTGROUP1",  
> theXML\_TEST\_ID: "10",  
> theXML\_TEST\_NAME: "VOLTAGETEST",  
> theXML\_TEST\_VALUE: "2\_34523",  
> theXML\_TEST\_STATUS: "OK"  
> }

Edit:  
I did it using mutate:

> add\_field =\> { "TEST\_ID" =\> "%{[theXML][PRODUCT][0][GROUP][TEST][ID]}" }  
> add\_field =\> { "TEST\_NAME" =\> "%{[theXML][PRODUCT][0][GROUP][TEST][NAME]}" }  
> add\_field =\> { "TEST\_VALUE" =\> "%{[theXML][PRODUCT][0][GROUP][TEST][VALUE]}" }  
> add\_field =\> { "TEST\_STATUS" =\> "%{[theXML][PRODUCT][0][GROUP][TEST][STATUS]}" }

Result in elastic:  
 ![2019-04-18%2015_51_12-Discover%20-%20Kibana](https://us1.discourse-cdn.com/elastic/original/3X/e/2/e2b7abb08661127d4b53c1da6399d0efed9182e2.png)

Thanks again @Badger.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 16, 2019, 1:20pm UTC](https://discuss.elastic.co/t/turning-one-xml-into-multiple-events-where-not-all-values-are-always-filled/176384/6 "2019-05-16T13:20:06Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.