Logstash, split event from an xml file in multiples documents keeping information from root tags

Elastic Stack Logstash
1

Parsing xml file via logstash. The documents should be parsed in a flattened structure.
Example:

<?xml version="1.0" encoding="UTF-8"?>
<root number="34">
	<eventlist>
	       <event name="hey" />
		<event name="you" />
	</eventlist>
</root>

The output is required as below:
{
"number":"34"
"name": "Hey"
}
{
"number":"34"
"name": "you"
}

Please help on this on how the configuration file in logstash should be written . Thanks in advance.

2

Please edit your post and format the XML snippet as preformatted text so that we can read what the input XML looks like.

3

This XML example is from stackoverflow.com. I have an xml similar to this. I just want to parse the xml and store in a flattened structure in elasticsearch.

4

You'll want to use the split filter on the [eventlist][event] array field produced by the xml filter. That should produce multiple events where the only difference is "hey" and "you". I'm getting issues with the plugins that ship with LS 2.3.2 but I suspect it should work with LS 2.4.0. Something like this should be close to what you want:

filter {
  xml {
    source => "message"
    store_xml => true
    target => ""
  }
  split {
    field => "[eventlist][0][events]"
  }
}

(Well, you'll afterwards have to perform various manipulations to get exactly what you want but those are simple changes.)

1 Like
5

Thanks.

But I get the below error when I use:

split {
field => "[eventlist][0][events]"
}

LogStash::ConfigurationError: Only String and Array types are splittable. field:
[eventlist][0][events] is of type = NilClass

6

With Logstash 2.4.0?

7

I'm using Logstash 2.2.2

8

I don't expect my example to work with anything prior to LS 2.4.0.

9

Oh ok. PFB the XML I want to parse:

<R browserExecutionType="Parallel" endTime="02/09/2016 04:21:27 PM" executionEnvironment="Local" name="Test Execution R - bookHotelNew" startTime="02/09/2016 04:18:46 PM">
<TS TSExecutionType="Sequence" browser="CHROME" desc="bookHotelNew_CHROME" endTime="02/09/2016 04:19:54 PM" iterationType="TestScenario" name="bookHotelNew_CHROME" startTime="02/09/2016 04:18:46 PM" status="0">
	<TC desc="TS_bookHotelNew" endTime="02/09/2016 04:19:46 PM" name="TS_bookHotelNew" startTime="02/09/2016 04:18:50 PM" status="0">
		<TCIter endTime="02/09/2016 04:19:46 PM" startTime="02/09/2016 04:18:51 PM" status="0" value="0">
			<PB bddKeyWord="" desc="BPC_s02_Login" endTime="02/09/2016 04:19:03 PM" name="BPC_s02_Login" startTime="02/09/2016 04:18:51 PM" status="1" totalEnabledSteps="4" totalSteps="4">
				<BPIter endTime="02/09/2016 04:19:03 PM" failed="0" notExecuted="0" passed="0" skipped="0" startTime="02/09/2016 04:18:51 PM" status="1" totalEnabledSteps="4" totalSteps="4" value="0">
					<Res ErrorImagePath="" actSeq="1" action="driver_get" conditionString="NA" conditionType="NA" endTime="02/09/2016 04:18:54 PM" fromStep="0" isConditionApplied="false" iterations="0" message="Perform 'driver_get' operation for the value 'http://adactin.com//HotelApp/index.php,'" object="" outputParamName="" outputParamValue="" skipReason="" startTime="02/09/2016 04:18:51 PM" status="1" step="0" toStep="0" />
					<Res ErrorImagePath="" actSeq="2" action="element_setElementText" conditionString="NA" conditionType="NA" data="andisrinu," endTime="02/09/2016 04:18:57 PM" fromStep="0" isConditionApplied="false" iterations="0" message="Perform 'element_setElementText' operation on 'By.id: username' for the value 'andisrinu,'" object="By.id: username" outputParamName="" outputParamValue="" skipReason="" startTime="02/09/2016 04:18:54 PM" status="1" step="0" toStep="0" />

				</BPIter>
			</PB>
		</TCIter>
	</TC>
	<TC desc="TS_testSc" endTime="02/09/2016 04:19:54 PM" name="TS_testSc" startTime="02/09/2016 04:19:46 PM" status="1">
		<TCIter endTime="02/09/2016 04:19:54 PM" startTime="02/09/2016 04:19:46 PM" status="1" value="0">
			<PB bddKeyWord="" desc="BPC_s01_Login_Logout" endTime="02/09/2016 04:19:54 PM" name="BPC_s01_Login_Logout" startTime="02/09/2016 04:19:46 PM" status="1" totalEnabledSteps="6" totalSteps="6">
				<BPIter endTime="02/09/2016 04:19:54 PM" failed="0" notExecuted="0" passed="0" skipped="0" startTime="02/09/2016 04:19:46 PM" status="1" totalEnabledSteps="6" totalSteps="6" value="0">
					<Res ErrorImagePath="" actSeq="1" action="driver_get" conditionString="NA" conditionType="NA" endTime="02/09/2016 04:19:46 PM" fromStep="0" isConditionApplied="false" iterations="0" message="Perform 'driver_get' operation for the value 'http://adactin.com//HotelApp/index.php,'" object="" outputParamName="" outputParamValue="" skipReason="" startTime="02/09/2016 04:19:46 PM" status="1" step="0" toStep="0" />
				</BPIter>
			</PB>
		</TCIter>
	</TC>
</TS>
<TS TSExecutionType="Sequence" browser="IE" desc="bookHotelNew_IE" endTime="02/09/2016 04:21:26 PM" iterationType="TestScenario" name="bookHotelNew_IE" startTime="02/09/2016 04:19:58 PM" status="0">
	<TC desc="TS_bookHotelNew" endTime="02/09/2016 04:21:06 PM" name="TS_bookHotelNew" startTime="02/09/2016 04:20:00 PM" status="0">
		<TCIter endTime="02/09/2016 04:21:06 PM" startTime="02/09/2016 04:20:00 PM" status="0" value="0">
			<PB bddKeyWord="" desc="BPC_s02_Login" endTime="02/09/2016 04:20:08 PM" name="BPC_s02_Login" startTime="02/09/2016 04:20:00 PM" status="1" totalEnabledSteps="4" totalSteps="4">
				<BPIter endTime="02/09/2016 04:20:08 PM" failed="0" notExecuted="0" passed="0" skipped="0" startTime="02/09/2016 04:20:00 PM" status="1" totalEnabledSteps="4" totalSteps="4" value="0">
					<Res ErrorImagePath="" actSeq="1" action="driver_get" conditionString="NA" conditionType="NA" endTime="02/09/2016 04:20:00 PM" fromStep="0" isConditionApplied="false" iterations="0" message="Perform 'driver_get' operation for the value 'http://adactin.com//HotelApp/index.php,'" object="" outputParamName="" outputParamValue="" skipReason="" startTime="02/09/2016 04:20:00 PM" status="1" step="0" toStep="0" />
				</BPIter>
			</PB>
		</TCIter>
	</TC>
</TS>

So i want to parse in Logstash in such a way that i can store in a flatten data structure in ES.
Any help on this would be great.