HI Team,
I'm just working on new type of XML file. I have some array information in that file i want to gather all those data.. This array fields are repeating with same tags due to this while processing the log the i'm getting empty array value and help me to concat all the value and store in a field. Please find the sample code for your refrerence
<_5:TelephoneCommunication>
<5:ChannelCode>Day</5:ChannelCode>
<5:CompleteNumber>312</5:CompleteNumber>
<5:UseCode>Day</5:UseCode>
<_5:TelephoneCommunication>
<5:ChannelCode>Evening</5:ChannelCode>
<5:CompleteNumber>123</5:CompleteNumber>
<5:UseCode>Evening</5:UseCode>
<_5:TelephoneCommunication>
<5:ChannelCode>Cell</5:ChannelCode>
<5:CompleteNumber>231</5:CompleteNumber>
<5:UseCode>Cell</5:UseCode>
xml code to extract the info,
"//body/ProcessMessageResponse/payload/content/ConfirmBOD/ConfirmBODDataArea/RepairOrder/RepairOrderHeader/OwnerParty/SpecifiedPerson/TelephoneCommunication/ChannelCode/text()","user_call_mode",
What do the events you're currently getting look like? How would you like them to look instead?
HI magnus,
Actually im getting empty value on that field instead of that i want to get the repeated value in the array . For example,
you can find the field "ChannelCode" thrice in above my example, i want to capture all the three value in to respective field. This is what i want to do it.
currently getting
user_call_mode :[]
Expected Result
user_call_mode:"Day|Evening|Cell"
If the XML element begin with "5:" I'd expect you to have to include that in the XPath expression. If you provide the actual XML file (or parts of it) and the exact xml filter you use it'll be easier to help out. Make sure you format both as preformatted text when posting so that the forum software doesn't mangle the text.
what i mentioned is exact xml and when it not comes in array format its working fine and im getting event.
like this
<5:ChannelCode>Cell
<5:CompleteNumber>231
<5:UseCode>Cell
when my input is repeat more than once im facing this issue
What you posted contains no closing tags so it can't possibly be the exact same as your input. I'm trying to reproduce your problem but then I need to know exactly what your input looks like. You also haven't posted exactly what the resulting event looks like.
In the below sample i need to capture all the description text into one field how could i process it, Please find the below input
<_9:BODSuccessMessage>
<_9:WarningProcessMessage>
<_9:Description>/p:ProcessRepairOrderType[1]/p:ProcessRepairOrderDataArea[1]/p:RepairOrder[1]/p:Job[2]/p:OperationName[1] | Length must be up to 80 characters =>Node: OperationName | RO063</_9:Description>
<_9:ReasonCode>Invalid Required Value
</_9:ReasonCode>
</_9:WarningProcessMessage>
<_9:WarningProcessMessage>
<_9:Description>/p:ProcessRepairOrderType[1]/p:ProcessRepairOrderDataArea[1]/p:RepairOrder[1]/p:Job[2]/p:OperationName[1] | Length must be up to 80 characters =>Node: OperationName | RO063</_9:Description>
<_9:ReasonCode>Invalid Required Value</_9:ReasonCode>
</_9:WarningProcessMessage>
<_9:WarningProcessMessage>
<_9:Description>/p:ProcessRepairOrderType[1]/p:ProcessRepairOrderDataArea[1]/p:RepairOrder[1]/p:Job[2]/p:CodesAndCommentsExpanded[1]/p:TechnicianNotes[1] | Format Error. Length must be up to 255 characters =>Node: TechnicianNotes | RO072</_9:Description>
<_9:ReasonCode>Invalid Required Value</_9:ReasonCode>
</_9:WarningProcessMessage>
<_9:WarningProcessMessage>
<_9:Description>/p:ProcessRepairOrderType[1]/p:ProcessRepairOrderDataArea[1]/p:RepairOrder[1]/p:Job[2]/p:ServiceLabor[1]/p:LaborOperationDescription[1] | Length must be up to 80 characters =>Node: LaborOperationDescription | RO087</_9:Description>
<_9:ReasonCode>Invalid Required Value</_9:ReasonCode>
</_9:WarningProcessMessage>
</_9:BODSuccessMessage>
This appears to work:
filter {
xml {
store_xml => false
source => "message"
xpath => ["//*[local-name()='Description']/text()", "description"]
}
}
This is clumsy and will not work if you have other Description elements in your document besides these in the "_9" namespace. It seems the xml filter allows you to register namespaces with the namespaces option, which would be preferable and would allow you to write a more exact XPath expression.
HI Magnus,
i tried above method also but it gives same result for me , which i got earlier.
Please give a complete reproducible recipe for your problem, i.e. an input string, a complete configuration, and the output you get.
Ok please find the input and config detail,
Input:
<?xml version="1.0" encoding="UTF-8"?>
<p:ServiceMessageObject >
<_5:ConfirmBOD releaseID="5.4.4">
<_5:ConfirmBODDataArea>
<_5:BOD>
<_9:BODSuccessMessage>
<_9:WarningProcessMessage>
<_9:Description>/p:ProcessRepairOrderType[1]/p:ProcessRepairOrderDataArea[1]/p:RepairOrder[1]/p:Job[1]/p:CodesAndCommentsExpanded[1]/p:TechnicianNotes[1] | Format Error. Length must be up to 255 characters =>Node: TechnicianNotes | RO072
<_9:ReasonCode>Invalid Required Value
<_9:WarningProcessMessage>
<_9:Description>/p:ProcessRepairOrderType[1]/p:ProcessRepairOrderDataArea[1]/p:RepairOrder[1]/p:Job[2]/p:OperationName[1] | Length must be up to 80 characters =>Node: OperationName | RO063
<_9:ReasonCode>Invalid Required Value
<_9:WarningProcessMessage>
<_9:Description>/p:ProcessRepairOrderType[1]/p:ProcessRepairOrderDataArea[1]/p:RepairOrder[1]/p:Job[2]/p:CodesAndCommentsExpanded[1]/p:TechnicianNotes[1] | Format Error. Length must be up to 255 characters =>Node: TechnicianNotes | RO072
<_9:ReasonCode>Invalid Required Value
<_9:WarningProcessMessage>
<_9:Description>/p:ProcessRepairOrderType[1]/p:ProcessRepairOrderDataArea[1]/p:RepairOrder[1]/p:Job[2]/p:ServiceLabor[1]/p:LaborOperationDescription[1] | Length must be up to 80 characters =>Node: LaborOperationDescription | RO087
<_9:ReasonCode>Invalid Required Value
Config:
xml{
source => "message"
store_xml => false
#force_array => true
xpath => [
"//*[local-name()='Description']/text()","response_desc",
"//body/ProcessMessageResponse/payload/content/ConfirmBOD/ConfirmBODDataArea/BOD/BODSuccessMessage/WarningProcessMessage/ReasonCode/text()","response_reason"
]
}
I asked for
- an input string,
- a complete configuration, and
- the output you get
and I got
- a garbled input string (format all logs, configuration, and any XML as preformatted text with the
</> toolbar button when posting),
- incomplete configuration (no inputs or outputs), and
- no output.
Please try again. I'm trying to save us all time here.
Please find my entire file details,
input file:
indent preformatted text by 4 spaces
Config file:
input {
beats{
port => 10523
}
}
filter
{
if [type] == "ddoa_res" {
xml{
source => "message"
store_xml => false
#force_array => true
xpath => [
"//context/correlation/fileName/text()","response_fileName",
"//body/ProcessMessageResponse/payload/content/ConfirmBOD/@releaseID","app_res_releaseID",
"//body/ProcessMessageResponse/payload/content/ConfirmBOD/ApplicationArea/Sender/CreatorNameCode/text()","app_res_creatorNameCode",
"//body/ProcessMessageResponse/payload/content/ConfirmBOD/ApplicationArea/Sender/SenderNameCode/text()","app_res_senderNameCode",
"//body/ProcessMessageResponse/payload/content/ConfirmBOD/ApplicationArea/Sender/SystemVersion/text()","app_res_sysVersion",
"//body/ProcessMessageResponse/payload/content/ConfirmBOD/ApplicationArea/CreationDateTime/text()","app_res_creationDateTime",
"//body/ProcessMessageResponse/payload/content/ConfirmBOD/ApplicationArea/BODID/text()","app_res_bodID",
"//body/ProcessMessageResponse/payload/content/ConfirmBOD/ApplicationArea/Destination/DestinationNameCode/text()","app_res_destinationName",
"CONCAT(//*[local-name()='Description']/text()|)","response_desc",
"//body/ProcessMessageResponse/payload/content/ConfirmBOD/ConfirmBODDataArea/BOD/BODSuccessMessage/WarningProcessMessage/ReasonCode/text()","response_reason"
]
}
mutate{
remove_field => ["message","type"]
}
fingerprint {
source => ["response_fileName"]
target => "fingerprint"
key => "78787878"
method => "SHA1"
concatenate_sources => true
}
}
}
output{
elasticsearch {
#index => "logstash-dd.ddoa_req_log_v1"
hosts => ["xxxxxx:9200"]
document_id => "%{fingerprint}" # !!! prevent duplication
}
stdout {
codec => rubydebug
}
}
The XML markup you posted is still garbled because you didn't format it correctly.
how to format it could you tell me that i ll change according that
I said: format all logs, configuration, and any XML as preformatted text with the </> toolbar button when posting.
<?xml version="1.0" encoding="UTF-8"?>
<p:ServiceMessageObject xsi:type="p:ServiceMessageObject" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:_1="http://www.openapplications.org/oagis/9/unqualifieddatatypes/1.1" xmlns:_5="http://www.starstandard.org/STAR/5" xmlns:_9="http://www.openapplications.org/oagis/9" xmlns:bi="wsdl.http://www.starstandards.org/webservices/2009/transport/bindings" xmlns:m="http://MA_STARWS_P2" xmlns:tr="http://www.starstandards.org/webservices/2009/transport" xmlns:p="http://www.ibm.com/websphere/sibx/smo/v6.0.1">
<context>
<correlation xsi:type="m:CEIAuditMetadata">
<dealerID xsi:type="_1:IdentifierType">04136</dealerID>
</correlation>
</context>
<headers>
<SMOHeader>
<MessageUUID>E7B4A942-0159-4000-E000-612E0A354DF3</MessageUUID>
<Version>
<Version>7</Version>
<Release>0</Release>
<Modification>0</Modification>
</Version>
<MessageType>Response</MessageType>
</SMOHeader>
<SOAPHeader>
<nameSpace>http://www.starstandards.org/webservices/2009/transport</nameSpace>
<name>payloadManifest</name>
<prefix>tran</prefix>
<value xsi:type="tr:PayloadManifest">
<tr:manifest contentID="bod1" element="ConfirmBOD"/>
</value>
</SOAPHeader>
</headers>
<body xsi:type="bi:ProcessMessageOut">
<tr:ProcessMessageResponse>
<tr:payload>
<tr:content id="bod1"><_5:ConfirmBOD releaseID="5.4.4">
<_5:ApplicationArea>
<_5:Sender>
<_5:CreatorNameCode>TY</_5:CreatorNameCode>
<_5:SenderNameCode>DDOA</_5:SenderNameCode>
<_5:DealerNumberID>04136</_5:DealerNumberID>
<_5:SystemVersion>TY2.0</_5:SystemVersion>
</_5:Sender>
<_5:CreationDateTime>2017-01-29T00:51:33Z</_5:CreationDateTime>
<_5:BODID>28c4f0e6-deac-364f-98ed-04cda62a1a4c</_5:BODID>
<_5:Destination>
<_5:DestinationNameCode>CORA-6 C2.02</_5:DestinationNameCode>
<_5:DestinationSoftware>TZ523</_5:DestinationSoftware>
</_5:Destination>
</_5:ApplicationArea>
<_5:ConfirmBODDataArea>
<_5:Confirm>
<_5:OriginalApplicationArea>
<_5:Sender>
<_5:CreatorNameCode>AD</_5:CreatorNameCode>
<_5:SenderNameCode>CORA-6 C2.02</_5:SenderNameCode>
<_5:DealerNumberID>04136</_5:DealerNumberID>
<_5:SystemVersion>TZ523</_5:SystemVersion>
</_5:Sender>
<_5:CreationDateTime>2017-01-29T00:51:26Z</_5:CreationDateTime>
<_5:BODID>59c51225-d5ad-3857-9fc0-65be22716a4e</_5:BODID>
<_5:Destination>
<_5:DestinationNameCode>TY</_5:DestinationNameCode>
</_5:Destination>
</_5:OriginalApplicationArea>
</_5:Confirm>
<_5:BOD>
<_9:BODSuccessMessage>
<_9:WarningProcessMessage>
<_9:Description>/p:ProcessRepairOrderType[1]/p:ProcessRepairOrderDataArea[1]/p:RepairOrder[1]/p:Job[1]/p:CodesAndCommentsExpanded[1]/p:TechnicianNotes[1] | Format Error. Length must be up to 255 characters =>Node: TechnicianNotes | RO072</_9:Description>
<_9:ReasonCode>Invalid Required Value</_9:ReasonCode>
</_9:WarningProcessMessage>
<_9:WarningProcessMessage>
<_9:Description>/p:ProcessRepairOrderType[1]/p:ProcessRepairOrderDataArea[1]/p:RepairOrder[1]/p:Job[2]/p:OperationName[1] | Length must be up to 80 characters =>Node: OperationName | RO063</_9:Description>
<_9:ReasonCode>Invalid Required Value</_9:ReasonCode>
</_9:WarningProcessMessage>
</_9:BODSuccessMessage>
</_5:BOD>
</_5:ConfirmBODDataArea>
</_5:ConfirmBOD></tr:content>
</tr:payload>
</tr:ProcessMessageResponse>
</body>
</p:ServiceMessageObject>s
input {
beats{
port => 10523
}
}
filter
{
if [type] == "ddoa_res" {
xml{
source => "message"
store_xml => false
#force_array => true
xpath => [
"//context/correlation/fileName/text()","response_fileName",
"//body/ProcessMessageResponse/payload/content/ConfirmBOD/@releaseID","app_res_releaseID",
"//body/ProcessMessageResponse/payload/content/ConfirmBOD/ApplicationArea/Sender/CreatorNameCode/text()","app_res_creatorNameCode",
"//body/ProcessMessageResponse/payload/content/ConfirmBOD/ApplicationArea/Sender/SenderNameCode/text()","app_res_senderNameCode",
"//body/ProcessMessageResponse/payload/content/ConfirmBOD/ApplicationArea/Sender/SystemVersion/text()","app_res_sysVersion",
"//body/ProcessMessageResponse/payload/content/ConfirmBOD/ApplicationArea/CreationDateTime/text()","app_res_creationDateTime",
"//body/ProcessMessageResponse/payload/content/ConfirmBOD/ApplicationArea/BODID/text()","app_res_bodID",
"//body/ProcessMessageResponse/payload/content/ConfirmBOD/ApplicationArea/Destination/DestinationNameCode/text()","app_res_destinationName",
"CONCAT(//*[local-name()='Description']/text()|)","response_desc",
"//body/ProcessMessageResponse/payload/content/ConfirmBOD/ConfirmBODDataArea/BOD/BODSuccessMessage/WarningProcessMessage/ReasonCode/text()","response_reason"
]
}
mutate{
remove_field => ["message","type"]
}
fingerprint {
source => ["response_fileName"]
target => "fingerprint"
key => "78787878"
method => "SHA1"
concatenate_sources => true
}
}
}
output{
elasticsearch {
#index => "logstash-dd.ddoa_req_log_v1"
hosts => ["xxxxxx:9200"]
document_id => "%{fingerprint}" # !!! prevent duplication
}
stdout {
codec => rubydebug
}
}
At least my Logstash complains about the CONCAT stuff in the XPath expression. When I remove that and return to what I gave you earlier, it works as expected:
...
"response_desc" => [
[0] "/p:ProcessRepairOrderType[1]/p:ProcessRepairOrderDataArea[1]/p:RepairOrder[1]/p:Job[1]/p:CodesAndCommentsExpanded[1]/p:TechnicianNotes[1] | Format Error. Length must be up to 255 characters =>Node: TechnicianNotes | RO072",
[1] "/p:ProcessRepairOrderType[1]/p:ProcessRepairOrderDataArea[1]/p:RepairOrder[1]/p:Job[2]/p:OperationName[1] | Length must be up to 80 characters =>Node: OperationName | RO063"
]
...
I'm out of patience. This is my last response in this thread.
