Log Pipeline

[marchias (Sean Marchiafava)]

I've got a non-standard log on my hands that I'm trying to create a custom ingest pipeline for. I can get the first section up to the "Action" key using Grok like below.

%{CISCOTAG:date}\s*%{HOUR:hour}:%{MINUTE:minute}:%{SECOND:second}\s*%{WORD:type}\s*%{USER:user}

The second section with all the Key="Value" pairs I really have not idea what to do. I tried a KV processor and it only works on the first pair. Anyone have an idea on a good way to tackle the example below:

[{
"_source": {
"event": {
"original": "2022-09-01 23:16:19 notice user Sep 1 23:16:19 Sep authpriv.notice 01: 23:16:17--5:00 192.168.1.1 Action="drop" inzone="Internal" service_id="Any_UDP" src="172.16.4.16" dst="224.0.0.252" proto="17" user="" src_user_name="" src_machine_name="" src_user_dn="" snid="" dst_user_name="" dst_machine_name="" dst_user_dn="" UP_match_table="TABLE_START" ROW_START="0" match_id="13" layer_uuid="9fced3b3-5da9-494d-b7f1-3242694d99f8" layer_name="internal" rule_uid="00000780-0000-0000-0000-000000000000" rule_name="Incoming/Internal Default Policy" ROW_END="0" UP_match_table="TABLE_END" UP_action_table="TABLE_START" ROW_START="0" action="0" ROW_END="0" UP_action_table="TABLE_END" ProductName="VPN-1 & FireWall-1" svc="5355" sport_svc="56210" ProductFamily="56210""

	}
}

}]

[stephenb (Stephen Brown)]

Get the first part parsed then put the KV stuff into %{GREEDYDATA:message_details}

Then this should work... I would like to tell you I figured this out on my own ... but I looked in 1 of our FW parsers and it was close

"field_split": """ (?=[a-z\_\-]+=)""",

But you have Upper Case too so with a RegEx Debugger I got there
"field_split": """ (?=[a-zA-Z\_\-]+=)""",

You also have some odd duplicates so it made arrays... that is cool did not even know it could do that.

So hopefully this gets you close

POST _ingest/pipeline/_simulate
{
  "pipeline": {
    "processors": [
      {
          "kv": {
          "field": "message_details",
          "field_split": """ (?=[a-zA-Z\_\-]+=)""",
          "value_split": "=",
          "ignore_missing": true,
          "ignore_failure": false,
          "trim_value": "\"",
          "strip_brackets": true
        }
      }
    ]
  },
  "docs": [
    {
      "_source": {
       "message_details": """Action="drop" inzone="Internal" service_id="Any_UDP" src="172.16.4.16" dst="224.0.0.252" proto="17" user="" src_user_name="" src_machine_name="" src_user_dn="" snid="" dst_user_name="" dst_machine_name="" dst_user_dn="" UP_match_table="TABLE_START" ROW_START="0" match_id="13" layer_uuid="9fced3b3-5da9-494d-b7f1-3242694d99f8" layer_name="internal" rule_uid="00000780-0000-0000-0000-000000000000" rule_name="Incoming/Internal Default Policy" ROW_END="0" UP_match_table="TABLE_END" UP_action_table="TABLE_START" ROW_START="0" action="0" ROW_END="0" UP_action_table="TABLE_END" ProductName="VPN-1 & FireWall-1" svc="5355" sport_svc="56210" ProductFamily="56210"""
      }
    }
  ]
}


#Result
{
  "docs": [
    {
      "doc": {
        "_index": "_index",
        "_id": "_id",
        "_version": "-3",
        "_source": {
          "ProductFamily": "56210",
          "Action": "drop",
          "snid": "",
          "dst": "224.0.0.252",
          "dst_machine_name": "",
          "ProductName": "VPN-1 & FireWall-1",
          "ROW_START": [
            "0",
            "0"
          ],
          "src_user_name": "",
          "rule_uid": "00000780-0000-0000-0000-000000000000",
          "layer_uuid": "9fced3b3-5da9-494d-b7f1-3242694d99f8",
          "src_machine_name": "",
          "src_user_dn": "",
          "dst_user_name": "",
          "service_id": "Any_UDP",
          "action": "0",
          "layer_name": "internal",
          "svc": "5355",
          "dst_user_dn": "",
          "inzone": "Internal",
          "src": "172.16.4.16",
          "rule_name": "Incoming/Internal Default Policy",
          "match_id": "13",
          "UP_match_table": [
            "TABLE_START",
            "TABLE_END"
          ],
          "ROW_END": [
            "0",
            "0"
          ],
          "proto": "17",
          "UP_action_table": [
            "TABLE_START",
            "TABLE_END"
          ],
          "message_details": """Action="drop" inzone="Internal" service_id="Any_UDP" src="172.16.4.16" dst="224.0.0.252" proto="17" user="" src_user_name="" src_machine_name="" src_user_dn="" snid="" dst_user_name="" dst_machine_name="" dst_user_dn="" UP_match_table="TABLE_START" ROW_START="0" match_id="13" layer_uuid="9fced3b3-5da9-494d-b7f1-3242694d99f8" layer_name="internal" rule_uid="00000780-0000-0000-0000-000000000000" rule_name="Incoming/Internal Default Policy" ROW_END="0" UP_match_table="TABLE_END" UP_action_table="TABLE_START" ROW_START="0" action="0" ROW_END="0" UP_action_table="TABLE_END" ProductName="VPN-1 & FireWall-1" svc="5355" sport_svc="56210" ProductFamily="56210""",
          "sport_svc": "56210",
          "user": ""
        },
        "_ingest": {
          "timestamp": "2022-10-07T22:37:54.811937741Z"
        }
      }
    }
  ]
}