HI guys,
I'm trying to create a logstash pipeline that parses incoming CEF logs, apply some logic and then outputs the log in JSON format to the console. Some logs are a bit complicated to parse since the key=value pairs cannot always be recognized by a simple delimiter.
Take the following log for example: (This is parsed by a grok filter into default CEF keys and extensions)
"message" => "<14>Apr 27 00:38:45 paloaltovm CEF:0|PaloAltoNetworks|PAN-OS|11.0.1|Succeeded|CONFIG|1|rt=Apr 26 2023 22:38:44 GMT deviceExternalId=63CCFFBBE1C0D07 shost=82.217.2.18 cs3= act=set duser=PaloAlto destinationServiceName=Web msg= vsys vsys1 rulebase security rules blabla externalId=7226424316814950417 PanOSDGl1=0 PanOSDGl2=0 PanOSDGl3=0 PanOSDGl4=0 PanOSVsysName= dvchost=paloaltovm PanOSActionFlags=0x0 cs1Label=\"Before Change Detail\" cs1={} cs2Label=\"After Change Detail\" cs2={blabla 55c6e493-a011-4556-ac9e-6f5318913d3d { to [ any ]; from [ any ]; source [ testadress ]; destination [ any ]; source-user [ any ]; category [ any ]; application [ any ]; service [ application-default ]; source-hip [ any ]; destination-hip [ any ]; tag [ tag1 ]; action allow; rule-type universal; description yoyo!; } } PanOSFWDeviceGroup=0 PanOSPolicyAuditComment=auditcommenting"
Some fields are easy to parse such as "PanOSDGl1" and "dvchost". But I would love to be able to give the field "cs2" the value:
"{blabla 55c6e493-a011-4556-ac9e-6f5318913d3d { to [ any ]; from [ any ]; source [ testadress ]; destination [ any ]; source-user [ any ]; category [ any ]; application [ any ]; service [ application-default ]; source-hip [ any ]; destination-hip [ any ]; tag [ tag1 ]; action allow; rule-type universal; description yoyo!; } }"
and field "msg" the value:
vsys vsys1 rulebase security rules blabla
I don't know a good way to handle this. I can configure the firewall (which sends the logs to Logstash) to place the value of the field "msg" in "{}", but I don't know if this will help since all other fields are parsed with the filter:
kv {
field_split => " "
value_split => "="