I have a custom log that has xml contents inside, I tried using the xpath attribute. however the xml fields are not captured while the grok parsing times out sometimes for large values however, the grok filter works well in the heroku debugger
10/5/2016 12:07:35 AM :
Error thrown is:
No Error
*************************************************************************
Request sent is:
<InventoryMgmtRequest xmlns="http://www.cps.com/Ecommerce/Worldwide/AvailabilityService/Schemas/InventoryMgmtRequest"><ns0:MsgHeader MessageType="" MsgDate="8/19/2016 10:46:10 PM" SystemOfOrigin="ISCS" xmlns:ns0="http://www.cps.com/Ecommerce/AvailabilityService/Schemas/InventoryMgmtRequest"><ns0:OrderID>SCAO247424</ns0:OrderID><ns0:ReservationID></ns0:ReservationID><ns0:CRD></ns0:CRD></ns0:MsgHeader><ns0:MsgBody xmlns:ns0="http://www.cps.com/Ecommerce/Worldwide/AvailabilityService/Schemas/InventryMgmtRqst"><ns0:Product Sku="PF2800XC" Qty="20" IsExpress="false" IsTangible="true" Region="LA" Country="MX"><ns0:ProdType></ns0:ProdType><ns0:LineItemNum>1</ns0:LineItemNum><ns0:JCID></ns0:JCID></ns0:Product><ns0:Product Sku="CXC" Qty="30"><ns0:ProdType></ns0:ProdType><ns0:LineItemNum>2</ns0:LineItemNum><ns0:JCID></ns0:JCID></ns0:Product><ns0:Product Sku="C12AC" Qty="30" IsExpress="false" IsTangible="true" Region="LA" Country="MX"><ns0:ProdType></ns0:ProdType><ns0:LineItemNum>3</ns0:LineItemNum><ns0:JCID></ns0:JCID></ns0:Product><ns0:Product Sku="CE4C" Qty="30" IsExpress="false" IsTangible="true"><ns0:ProdType></ns0:ProdType><ns0:LineItemNum>4</ns0:LineItemNum><ns0:JCID></ns0:JCID></ns0:Product><ns0:Product Sku="C11AC" Qty="30" IsExpress="false" IsTangible="true"><ns0:ProdType></ns0:ProdType><ns0:LineItemNum>5</ns0:LineItemNum><ns0:JCID></ns0:JCID></ns0:Product><ns0:Product Sku="CF281XC" Qty="50" IsExpress="false" IsTangible="true" Region="LA" Country="MX"><ns0:ProdType></ns0:ProdType><ns0:LineItemNum>6</ns0:LineItemNum><ns0:JCID></ns0:JCID></ns0:Product></ns0:MsgBody></InventoryMgmtRequest> *************************************************************************
I have been able to fetch this into a single line input from filebeat. Now I want to parse each of the xml tags into specific fields. Here is how the config file looks like:
mutate
{
gsub => ["message", "\n", ":"]
}
grok
{
# check that fields match your batchserver log settings
match => { "message" =>"%{DATE_US:logdate}\s%{TIME:logtime}\s?%{WORD:AM_PM}\s\:\s\:%{GREEDYDATA:errormessage}\:\:%{GREEDYDATA:errordata}\:%{GREEDYDATA:lineseparator}\:%{GREEDYDATA: requestmessage}\:\:%{GREEDYDATA:requestdata}\:%{GREEDYDATA:lineseparator1}\:%{GREEDYDATA:responsemessage}\:\:%{GREEDYDATA:responsedata}\:%{GREEDYDATA:lineseparator2}"}
}
xml
{
source => "message"
store_xml => "false"
xpath => ["/InventoryMgmtRequest/text()","url"]
#xpath => ["/InventoryMgmtRequest/ns0:MsgHeader/ns0:OrderID/text()","OrderID"]
#xpath => ["/InventoryMgmtRequest/ns0:MsgHeader/ns0:ReservationID/text()","ReservationID"]
#xpath => ["/InventoryMgmtRequest/ns0:MsgHeader/ns0:CRD/text()","CRD"]
#}
#mutate {
#remove_field => ["message"]
#remove_field => ["lineseparator"]
#remove_field => ["lineseparator1"]
replace =>
{
"url" =>"%{[url][0]}"
}
}