Logstash - XML parsing including nested objects

Elastic Stack Logstash
1

Problem - Unable to convert object fields to flat fields upon parsing the XML
Input - Dynamic Soap xml
Example -

<SOAP-ENV:Envelope xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/" xmlns:SOAP-ENC="http://schemas.xmlsoap.org/soap/encoding/" xmlns:m0="xmlns-m0.example.com" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><SOAP-ENV:Body><OrderRequest xmlns="xmlns.example.com" xsi:schemaLocation="xsi.example.com"><m0:standardHeader xsi:schemaLocation="xmlns-m0.example.com"><m0:e2e><m0:E2EDATA>19=1.1.1.1.1.1,15=APP2,16=APP1,13=Order Request,11=REQ,12=VOL-1234:Order Key : 123456789,2=Order,10=4ocdluj6bu,7=V_SubmitOrder,6=PCK002069,5=SubmitOrder,4=OneView,9=siebel,8=UNKNOWN,1.2=APP1234:4ocdm363lj:4ocdo8g1b6,1.1=APP1234:4ocdluj669:4ocdlujgw6</m0:E2EDATA></m0:e2e><m0:serviceState><m0:stateCode>OK</m0:stateCode></m0:serviceState><m0:serviceAddressing><m0:from>APP2</m0:from><m0:to><m0:address>APP1</m0:address></m0:to><m0:messageId>1</m0:messageId><m0:action>Order</m0:action></m0:serviceAddressing></m0:standardHeader><Order><OrderType>order</OrderType><ClientOrderKey>12345678</ClientOrderKey><CustomerId>12345678</CustomerId><CustomerName>MR XYZ</CustomerName><OrderNumber>VOL-1234</OrderNumber><OrderDate>2009-01-10T17:07:57</OrderDate><SalesChannel>V</SalesChannel><SalesPartner>001.01001.000</SalesPartner><ListOfOrderItem><OrderItem><ClientOrderItemKey>12345678</ClientOrderItemKey><ProductInstanceId>VOL-1234</ProductInstanceId><Action>Delete</Action><PromotionProductInstanceId>VOL-1234</PromotionProductInstanceId><StartDate>2009-03-08T00:00:00</StartDate><EndDate/><CustomerAgreedDate>2009-01-10T23:59:59</CustomerAgreedDate><ProductName>Service</ProductName><Quantity>1</Quantity><ProductCode>12345678</ProductCode><ProductClass>Service</ProductClass><ProductType>Product</ProductType><ServiceId>12345678</ServiceId><PriceType>One-Time</PriceType><PriceFrequency>Monthly</PriceFrequency><PriceListId>1234</PriceListId><PriceListVATType>Inclusive</PriceListVATType><ListPrice>0.0</ListPrice><AdjustedPrice>0.0</AdjustedPrice><BillAccountNumber>1234567890</BillAccountNumber><InstallationAddress><AddressId>1234</AddressId><AddressNADKey>1234567890</AddressNADKey><AddressFormat>XXXX</AddressFormat><AddressMatched>N</AddressMatched><BuildingName/><BuildingNumber>1</BuildingNumber><Thoroughfare>XYZ</Thoroughfare><PostTown>ABC</PostTown><PostalOutcode>111</PostalOutcode><PostalIncode>2222</PostalIncode><Country>GB</Country></InstallationAddress><NrcPaymentType>Bill</NrcPaymentType><ListOfOrderItemAttribute><ItemAttribute><Name>CompletionDate</Name><PreviousValue/><Value/><Action>Delete</Action></ItemAttribute><ItemAttribute><Name>supplierServiceId</Name><PreviousValue/><Value>1234</Value><Action>Delete</Action></ItemAttribute></ListOfOrderItemAttribute></OrderItem></ListOfOrderItem></Order></OrderRequest></SOAP-ENV:Body></SOAP-ENV:Envelope>

Filter -

filter {
      xml {
	    id => "xml-filter" 
		source => "message" 
		store_xml => "true" 
		force_array => "false" 
		target => "xml" 
		remove_namespaces => "true"
	  }
}

Actual Output -
resultant fields are objects .e.g.

xml.Body.submitBillingOrderRequest.Order.ClientOrderKey: value
xml.Body.submitBillingOrderRequest.Order.CustomerId: value
xml.Body.submitBillingOrderRequest.Order.CustomerName: value

Expected Output -
the objective is to flatten such object fields so the o/p should look like -

ClientOrderKey: value
CustomerId: value
CustomerName: value
2

I would do that in ruby. This is an example of flattening objects. This is an example of stripping a prefix from a field name.

You might want to use something like

newK = k.sub(/\.([^.]*$)/, "\1")
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.