Concat the XML array value in logstash

Ganesh2303 · January 31, 2017, 10:41am

HI Team,
I'm just working on new type of XML file. I have some array information in that file i want to gather all those data.. This array fields are repeating with same tags due to this while processing the log the i'm getting empty array value and help me to concat all the value and store in a field. Please find the sample code for your refrerence

<_5:TelephoneCommunication>
<5:ChannelCode>Day</5:ChannelCode>
<5:CompleteNumber>312</5:CompleteNumber>
<5:UseCode>Day</5:UseCode>

<_5:TelephoneCommunication>
<5:ChannelCode>Evening</5:ChannelCode>
<5:CompleteNumber>123</5:CompleteNumber>
<5:UseCode>Evening</5:UseCode>

<_5:TelephoneCommunication>
<5:ChannelCode>Cell</5:ChannelCode>
<5:CompleteNumber>231</5:CompleteNumber>
<5:UseCode>Cell</5:UseCode>
xml code to extract the info,

"//body/ProcessMessageResponse/payload/content/ConfirmBOD/ConfirmBODDataArea/RepairOrder/RepairOrderHeader/OwnerParty/SpecifiedPerson/TelephoneCommunication/ChannelCode/text()","user_call_mode",

magnusbaeck · February 1, 2017, 8:48pm

What do the events you're currently getting look like? How would you like them to look instead?

Ganesh2303 · February 3, 2017, 9:52am

HI magnus,
Actually im getting empty value on that field instead of that i want to get the repeated value in the array . For example,

you can find the field "ChannelCode" thrice in above my example, i want to capture all the three value in to respective field. This is what i want to do it.

currently getting
user_call_mode :[]
Expected Result
user_call_mode:"Day|Evening|Cell"

magnusbaeck · February 3, 2017, 10:31am

If the XML element begin with "5:" I'd expect you to have to include that in the XPath expression. If you provide the actual XML file (or parts of it) and the exact xml filter you use it'll be easier to help out. Make sure you format both as preformatted text when posting so that the forum software doesn't mangle the text.

Ganesh2303 · February 3, 2017, 11:30am

what i mentioned is exact xml and when it not comes in array format its working fine and im getting event.
like this
<5:ChannelCode>Cell
<5:CompleteNumber>231
<5:UseCode>Cell
when my input is repeat more than once im facing this issue

magnusbaeck · February 6, 2017, 6:35am

What you posted contains no closing tags so it can't possibly be the exact same as your input. I'm trying to reproduce your problem but then I need to know exactly what your input looks like. You also haven't posted exactly what the resulting event looks like.

Ganesh2303 · February 7, 2017, 9:41am

In the below sample i need to capture all the description text into one field how could i process it, Please find the below input

 <_9:BODSuccessMessage>
                  <_9:WarningProcessMessage>     
               
<_9:Description>/p:ProcessRepairOrderType[1]/p:ProcessRepairOrderDataArea[1]/p:RepairOrder[1]/p:Job[2]/p:OperationName[1] |  Length must be up to 80 characters =>Node: OperationName | RO063</_9:Description>
                    <_9:ReasonCode>Invalid Required Value
</_9:ReasonCode>
              </_9:WarningProcessMessage>

              <_9:WarningProcessMessage>
               <_9:Description>/p:ProcessRepairOrderType[1]/p:ProcessRepairOrderDataArea[1]/p:RepairOrder[1]/p:Job[2]/p:OperationName[1] |  Length must be up to 80 characters =>Node: OperationName | RO063</_9:Description>
                <_9:ReasonCode>Invalid Required Value</_9:ReasonCode>
              </_9:WarningProcessMessage>
              <_9:WarningProcessMessage>
                <_9:Description>/p:ProcessRepairOrderType[1]/p:ProcessRepairOrderDataArea[1]/p:RepairOrder[1]/p:Job[2]/p:CodesAndCommentsExpanded[1]/p:TechnicianNotes[1] |  Format Error. Length must be up to 255 characters =>Node: TechnicianNotes | RO072</_9:Description>
                <_9:ReasonCode>Invalid Required Value</_9:ReasonCode>
              </_9:WarningProcessMessage>
              <_9:WarningProcessMessage>
                <_9:Description>/p:ProcessRepairOrderType[1]/p:ProcessRepairOrderDataArea[1]/p:RepairOrder[1]/p:Job[2]/p:ServiceLabor[1]/p:LaborOperationDescription[1] |  Length must be up to 80 characters =>Node: LaborOperationDescription | RO087</_9:Description>
                <_9:ReasonCode>Invalid Required Value</_9:ReasonCode>
              </_9:WarningProcessMessage>
            </_9:BODSuccessMessage>

magnusbaeck · February 7, 2017, 10:15am

This appears to work:

filter {
  xml {
    store_xml => false
    source => "message"
    xpath => ["//*[local-name()='Description']/text()", "description"]
  }
}

This is clumsy and will not work if you have other Description elements in your document besides these in the "_9" namespace. It seems the xml filter allows you to register namespaces with the namespaces option, which would be preferable and would allow you to write a more exact XPath expression.

Ganesh2303 · February 8, 2017, 9:13am

HI Magnus,
i tried above method also but it gives same result for me , which i got earlier.

magnusbaeck · February 8, 2017, 9:14am

Please give a complete reproducible recipe for your problem, i.e. an input string, a complete configuration, and the output you get.

Ganesh2303 · February 8, 2017, 9:21am

Ok please find the input and config detail,

Input:

<?xml version="1.0" encoding="UTF-8"?>

<p:ServiceMessageObject >

<_5:ConfirmBOD releaseID="5.4.4"> <_5:ConfirmBODDataArea> <_5:BOD> <_9:BODSuccessMessage> <_9:WarningProcessMessage> <_9:Description>/p:ProcessRepairOrderType[1]/p:ProcessRepairOrderDataArea[1]/p:RepairOrder[1]/p:Job[1]/p:CodesAndCommentsExpanded[1]/p:TechnicianNotes[1] | Format Error. Length must be up to 255 characters =>Node: TechnicianNotes | RO072 <_9:ReasonCode>Invalid Required Value <_9:WarningProcessMessage> <_9:Description>/p:ProcessRepairOrderType[1]/p:ProcessRepairOrderDataArea[1]/p:RepairOrder[1]/p:Job[2]/p:OperationName[1] | Length must be up to 80 characters =>Node: OperationName | RO063 <_9:ReasonCode>Invalid Required Value <_9:WarningProcessMessage> <_9:Description>/p:ProcessRepairOrderType[1]/p:ProcessRepairOrderDataArea[1]/p:RepairOrder[1]/p:Job[2]/p:CodesAndCommentsExpanded[1]/p:TechnicianNotes[1] | Format Error. Length must be up to 255 characters =>Node: TechnicianNotes | RO072 <_9:ReasonCode>Invalid Required Value <_9:WarningProcessMessage> <_9:Description>/p:ProcessRepairOrderType[1]/p:ProcessRepairOrderDataArea[1]/p:RepairOrder[1]/p:Job[2]/p:ServiceLabor[1]/p:LaborOperationDescription[1] | Length must be up to 80 characters =>Node: LaborOperationDescription | RO087 <_9:ReasonCode>Invalid Required Value

Config:

xml{
source => "message"
store_xml => false
#force_array => true
xpath => [

			"//*[local-name()='Description']/text()","response_desc",
			"//body/ProcessMessageResponse/payload/content/ConfirmBOD/ConfirmBODDataArea/BOD/BODSuccessMessage/WarningProcessMessage/ReasonCode/text()","response_reason"
							
			
			]
	}

magnusbaeck · February 8, 2017, 9:36am

I asked for

an input string,
a complete configuration, and
the output you get

and I got

a garbled input string (format all logs, configuration, and any XML as preformatted text with the </> toolbar button when posting),
incomplete configuration (no inputs or outputs), and
no output.

Please try again. I'm trying to save us all time here.

Ganesh2303 · February 8, 2017, 10:45am

Please find my entire file details,

input file:
indent preformatted text by 4 spaces

Config file:

input {
    beats{
		port => 10523
		}
	
}

filter 
{
	
	if [type] == "ddoa_res" {
		xml{
			source => "message"
			store_xml => false 
			#force_array => true
			xpath => [				
				"//context/correlation/fileName/text()","response_fileName",					
				"//body/ProcessMessageResponse/payload/content/ConfirmBOD/@releaseID","app_res_releaseID",						
				"//body/ProcessMessageResponse/payload/content/ConfirmBOD/ApplicationArea/Sender/CreatorNameCode/text()","app_res_creatorNameCode",						
				"//body/ProcessMessageResponse/payload/content/ConfirmBOD/ApplicationArea/Sender/SenderNameCode/text()","app_res_senderNameCode",						
				"//body/ProcessMessageResponse/payload/content/ConfirmBOD/ApplicationArea/Sender/SystemVersion/text()","app_res_sysVersion",						
				"//body/ProcessMessageResponse/payload/content/ConfirmBOD/ApplicationArea/CreationDateTime/text()","app_res_creationDateTime",						
				"//body/ProcessMessageResponse/payload/content/ConfirmBOD/ApplicationArea/BODID/text()","app_res_bodID",						
				"//body/ProcessMessageResponse/payload/content/ConfirmBOD/ApplicationArea/Destination/DestinationNameCode/text()","app_res_destinationName",
				"CONCAT(//*[local-name()='Description']/text()|)","response_desc",
				"//body/ProcessMessageResponse/payload/content/ConfirmBOD/ConfirmBODDataArea/BOD/BODSuccessMessage/WarningProcessMessage/ReasonCode/text()","response_reason"
								
				
				]
		}
		
		
		
			mutate{
			remove_field => ["message","type"]
		}
		
		
					fingerprint {
                                source => ["response_fileName"]
                                target => "fingerprint"
                                key => "78787878"
                                method => "SHA1"
                                concatenate_sources => true
								}
										
	}
	
  }

output{

elasticsearch {		  
	  #index => "logstash-dd.ddoa_req_log_v1"		  
	  hosts => ["xxxxxx:9200"]				
			document_id => "%{fingerprint}" # !!! prevent duplication		  
	}
		
	
stdout {
			codec => rubydebug
		}

}

magnusbaeck · February 8, 2017, 10:50am

The XML markup you posted is still garbled because you didn't format it correctly.

Ganesh2303 · February 8, 2017, 10:53am

how to format it could you tell me that i ll change according that

magnusbaeck · February 8, 2017, 10:59am

I said: format all logs, configuration, and any XML as preformatted text with the </> toolbar button when posting.

Ganesh2303 · February 8, 2017, 11:07am

 <?xml version="1.0" encoding="UTF-8"?>
<p:ServiceMessageObject xsi:type="p:ServiceMessageObject" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:_1="http://www.openapplications.org/oagis/9/unqualifieddatatypes/1.1" xmlns:_5="http://www.starstandard.org/STAR/5" xmlns:_9="http://www.openapplications.org/oagis/9" xmlns:bi="wsdl.http://www.starstandards.org/webservices/2009/transport/bindings" xmlns:m="http://MA_STARWS_P2" xmlns:tr="http://www.starstandards.org/webservices/2009/transport" xmlns:p="http://www.ibm.com/websphere/sibx/smo/v6.0.1">
  <context>
    <correlation xsi:type="m:CEIAuditMetadata">
      <dealerID xsi:type="_1:IdentifierType">04136</dealerID>          
    </correlation>
  </context>
  <headers>
    <SMOHeader>
      <MessageUUID>E7B4A942-0159-4000-E000-612E0A354DF3</MessageUUID>
      <Version>
        <Version>7</Version>
        <Release>0</Release>
        <Modification>0</Modification>
      </Version>
      <MessageType>Response</MessageType>
    </SMOHeader>
    <SOAPHeader>
      <nameSpace>http://www.starstandards.org/webservices/2009/transport</nameSpace>
      <name>payloadManifest</name>
      <prefix>tran</prefix>
      <value xsi:type="tr:PayloadManifest">
        <tr:manifest contentID="bod1" element="ConfirmBOD"/>
      </value>
    </SOAPHeader>
  </headers>
  <body xsi:type="bi:ProcessMessageOut">
    <tr:ProcessMessageResponse>
      <tr:payload>
        <tr:content id="bod1"><_5:ConfirmBOD releaseID="5.4.4">
            <_5:ApplicationArea>
              <_5:Sender>
                <_5:CreatorNameCode>TY</_5:CreatorNameCode>
                <_5:SenderNameCode>DDOA</_5:SenderNameCode>
                <_5:DealerNumberID>04136</_5:DealerNumberID>
                <_5:SystemVersion>TY2.0</_5:SystemVersion>
              </_5:Sender>
              <_5:CreationDateTime>2017-01-29T00:51:33Z</_5:CreationDateTime>
              <_5:BODID>28c4f0e6-deac-364f-98ed-04cda62a1a4c</_5:BODID>
              <_5:Destination>
                <_5:DestinationNameCode>CORA-6 C2.02</_5:DestinationNameCode>
                <_5:DestinationSoftware>TZ523</_5:DestinationSoftware>
              </_5:Destination>
            </_5:ApplicationArea>
            <_5:ConfirmBODDataArea>
              <_5:Confirm>
                <_5:OriginalApplicationArea>
                  <_5:Sender>
                    <_5:CreatorNameCode>AD</_5:CreatorNameCode>
                    <_5:SenderNameCode>CORA-6 C2.02</_5:SenderNameCode>
                    <_5:DealerNumberID>04136</_5:DealerNumberID>
                    <_5:SystemVersion>TZ523</_5:SystemVersion>
                  </_5:Sender>
                  <_5:CreationDateTime>2017-01-29T00:51:26Z</_5:CreationDateTime>
                  <_5:BODID>59c51225-d5ad-3857-9fc0-65be22716a4e</_5:BODID>
                  <_5:Destination>
                    <_5:DestinationNameCode>TY</_5:DestinationNameCode>
                  </_5:Destination>
                </_5:OriginalApplicationArea>
              </_5:Confirm>
              <_5:BOD>
                <_9:BODSuccessMessage>
                  <_9:WarningProcessMessage>
                    <_9:Description>/p:ProcessRepairOrderType[1]/p:ProcessRepairOrderDataArea[1]/p:RepairOrder[1]/p:Job[1]/p:CodesAndCommentsExpanded[1]/p:TechnicianNotes[1] |  Format Error. Length must be up to 255 characters =>Node: TechnicianNotes | RO072</_9:Description>
                    <_9:ReasonCode>Invalid Required Value</_9:ReasonCode>
                  </_9:WarningProcessMessage>
                  <_9:WarningProcessMessage>
                    <_9:Description>/p:ProcessRepairOrderType[1]/p:ProcessRepairOrderDataArea[1]/p:RepairOrder[1]/p:Job[2]/p:OperationName[1] |  Length must be up to 80 characters =>Node: OperationName | RO063</_9:Description>
                    <_9:ReasonCode>Invalid Required Value</_9:ReasonCode>
                  </_9:WarningProcessMessage>
                
                  
                </_9:BODSuccessMessage>
              </_5:BOD>
            </_5:ConfirmBODDataArea>
          </_5:ConfirmBOD></tr:content>
      </tr:payload>
    </tr:ProcessMessageResponse>
  </body>
</p:ServiceMessageObject>s

Ganesh2303 · February 8, 2017, 11:07am

input {
    beats{
		port => 10523
		}
	
}

filter 
{
	
	if [type] == "ddoa_res" {
		xml{
			source => "message"
			store_xml => false 
			#force_array => true
			xpath => [				
				"//context/correlation/fileName/text()","response_fileName",					
				"//body/ProcessMessageResponse/payload/content/ConfirmBOD/@releaseID","app_res_releaseID",						
				"//body/ProcessMessageResponse/payload/content/ConfirmBOD/ApplicationArea/Sender/CreatorNameCode/text()","app_res_creatorNameCode",						
				"//body/ProcessMessageResponse/payload/content/ConfirmBOD/ApplicationArea/Sender/SenderNameCode/text()","app_res_senderNameCode",						
				"//body/ProcessMessageResponse/payload/content/ConfirmBOD/ApplicationArea/Sender/SystemVersion/text()","app_res_sysVersion",						
				"//body/ProcessMessageResponse/payload/content/ConfirmBOD/ApplicationArea/CreationDateTime/text()","app_res_creationDateTime",						
				"//body/ProcessMessageResponse/payload/content/ConfirmBOD/ApplicationArea/BODID/text()","app_res_bodID",						
				"//body/ProcessMessageResponse/payload/content/ConfirmBOD/ApplicationArea/Destination/DestinationNameCode/text()","app_res_destinationName",
				"CONCAT(//*[local-name()='Description']/text()|)","response_desc",
				"//body/ProcessMessageResponse/payload/content/ConfirmBOD/ConfirmBODDataArea/BOD/BODSuccessMessage/WarningProcessMessage/ReasonCode/text()","response_reason"
								
				
				]
		}
		
		
		
			mutate{
			remove_field => ["message","type"]
		}
		
		
					fingerprint {
                                source => ["response_fileName"]
                                target => "fingerprint"
                                key => "78787878"
                                method => "SHA1"
                                concatenate_sources => true
								}
										
	}
	
  }

output{

elasticsearch {		  
	  #index => "logstash-dd.ddoa_req_log_v1"		  
	  hosts => ["xxxxxx:9200"]				
			document_id => "%{fingerprint}" # !!! prevent duplication		  
	}
		
	
stdout {
			codec => rubydebug
		}

}

Ganesh2303 · February 8, 2017, 11:08am

Output result:

magnusbaeck · February 8, 2017, 11:16am

At least my Logstash complains about the CONCAT stuff in the XPath expression. When I remove that and return to what I gave you earlier, it works as expected:

...
    "response_desc" => [
        [0] "/p:ProcessRepairOrderType[1]/p:ProcessRepairOrderDataArea[1]/p:RepairOrder[1]/p:Job[1]/p:CodesAndCommentsExpanded[1]/p:TechnicianNotes[1] |  Format Error. Length must be up to 255 characters =&gt;Node: TechnicianNotes | RO072",
        [1] "/p:ProcessRepairOrderType[1]/p:ProcessRepairOrderDataArea[1]/p:RepairOrder[1]/p:Job[2]/p:OperationName[1] |  Length must be up to 80 characters =&gt;Node: OperationName | RO063"
    ]
...

I'm out of patience. This is my last response in this thread.