# Log Pipeline
**URL:** https://discuss.elastic.co/t/log-pipeline/315714
**Category:** Elastic Observability
**Tags:** ingest-pipeline
**Created:** [October 3, 2022, 6:49pm UTC](https://discuss.elastic.co/t/log-pipeline/315714 "2022-10-03T18:49:03Z")
**Posts on this page:** 3
**Page:** 1
### Author:  [@marchias](https://discuss.elastic.co/u/marchias)
#### Post date: [October 3, 2022, 6:49pm UTC](https://discuss.elastic.co/t/log-pipeline/315714/1 "2022-10-03T18:49:03Z")
I've got a non-standard log on my hands that I'm trying to create a custom ingest pipeline for. I can get the first section up to the "Action" key using Grok like below.
%{CISCOTAG:date}\s\*%{HOUR:hour}:%{MINUTE:minute}:%{SECOND:second}\s\*%{WORD:type}\s\*%{USER:user}
The second section with all the Key="Value" pairs I really have not idea what to do. I tried a KV processor and it only works on the first pair. Anyone have an idea on a good way to tackle the example below:
[{
"\_source": {
"event": {
"original": "2022-09-01 23:16:19 notice user Sep 1 23:16:19 Sep authpriv.notice 01: 23:16:17--5:00 192.168.1.1 Action="drop" inzone="Internal" service\_id="Any\_UDP" src="172.16.4.16" dst="224.0.0.252" proto="17" user="" src\_user\_name="" src\_machine\_name="" src\_user\_dn="" snid="" dst\_user\_name="" dst\_machine\_name="" dst\_user\_dn="" UP\_match\_table="TABLE\_START" ROW\_START="0" match\_id="13" layer\_uuid="9fced3b3-5da9-494d-b7f1-3242694d99f8" layer\_name="internal" rule\_uid="00000780-0000-0000-0000-000000000000" rule\_name="Incoming/Internal Default Policy" ROW\_END="0" UP\_match\_table="TABLE\_END" UP\_action\_table="TABLE\_START" ROW\_START="0" action="0" ROW\_END="0" UP\_action\_table="TABLE\_END" ProductName="VPN-1 & FireWall-1" svc="5355" sport\_svc="56210" ProductFamily="56210""
```
}
}
```
}]
---
### Author:  [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [October 7, 2022, 10:32pm UTC](https://discuss.elastic.co/t/log-pipeline/315714/2 "2022-10-07T22:32:04Z")
Get the first part parsed then put the KV stuff into `%{GREEDYDATA:message_details}`
Then this should work... I would like to tell you I figured this out on my own ... but I looked in 1 of our FW parsers and it was close
` "field_split": """ (?=[a-z\_\-]+=)""",`
But you have Upper Case too so with a RegEx Debugger I got there
` "field_split": """ (?=[a-zA-Z\_\-]+=)""",`
You also have some odd duplicates so it made arrays... that is cool did not even know it could do that.
So hopefully this gets you close
```auto
POST _ingest/pipeline/_simulate
{
"pipeline": {
"processors": [
{
"kv": {
"field": "message_details",
"field_split": """ (?=[a-zA-Z\_\-]+=)""",
"value_split": "=",
"ignore_missing": true,
"ignore_failure": false,
"trim_value": "\"",
"strip_brackets": true
}
}
]
},
"docs": [
{
"_source": {
"message_details": """Action="drop" inzone="Internal" service_id="Any_UDP" src="172.16.4.16" dst="224.0.0.252" proto="17" user="" src_user_name="" src_machine_name="" src_user_dn="" snid="" dst_user_name="" dst_machine_name="" dst_user_dn="" UP_match_table="TABLE_START" ROW_START="0" match_id="13" layer_uuid="9fced3b3-5da9-494d-b7f1-3242694d99f8" layer_name="internal" rule_uid="00000780-0000-0000-0000-000000000000" rule_name="Incoming/Internal Default Policy" ROW_END="0" UP_match_table="TABLE_END" UP_action_table="TABLE_START" ROW_START="0" action="0" ROW_END="0" UP_action_table="TABLE_END" ProductName="VPN-1 & FireWall-1" svc="5355" sport_svc="56210" ProductFamily="56210"""
}
}
]
}
#Result
{
"docs": [
{
"doc": {
"_index": "_index",
"_id": "_id",
"_version": "-3",
"_source": {
"ProductFamily": "56210",
"Action": "drop",
"snid": "",
"dst": "224.0.0.252",
"dst_machine_name": "",
"ProductName": "VPN-1 & FireWall-1",
"ROW_START": [
"0",
"0"
],
"src_user_name": "",
"rule_uid": "00000780-0000-0000-0000-000000000000",
"layer_uuid": "9fced3b3-5da9-494d-b7f1-3242694d99f8",
"src_machine_name": "",
"src_user_dn": "",
"dst_user_name": "",
"service_id": "Any_UDP",
"action": "0",
"layer_name": "internal",
"svc": "5355",
"dst_user_dn": "",
"inzone": "Internal",
"src": "172.16.4.16",
"rule_name": "Incoming/Internal Default Policy",
"match_id": "13",
"UP_match_table": [
"TABLE_START",
"TABLE_END"
],
"ROW_END": [
"0",
"0"
],
"proto": "17",
"UP_action_table": [
"TABLE_START",
"TABLE_END"
],
"message_details": """Action="drop" inzone="Internal" service_id="Any_UDP" src="172.16.4.16" dst="224.0.0.252" proto="17" user="" src_user_name="" src_machine_name="" src_user_dn="" snid="" dst_user_name="" dst_machine_name="" dst_user_dn="" UP_match_table="TABLE_START" ROW_START="0" match_id="13" layer_uuid="9fced3b3-5da9-494d-b7f1-3242694d99f8" layer_name="internal" rule_uid="00000780-0000-0000-0000-000000000000" rule_name="Incoming/Internal Default Policy" ROW_END="0" UP_match_table="TABLE_END" UP_action_table="TABLE_START" ROW_START="0" action="0" ROW_END="0" UP_action_table="TABLE_END" ProductName="VPN-1 & FireWall-1" svc="5355" sport_svc="56210" ProductFamily="56210""",
"sport_svc": "56210",
"user": ""
},
"_ingest": {
"timestamp": "2022-10-07T22:37:54.811937741Z"
}
}
}
]
}
```
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 4, 2022, 10:32pm UTC](https://discuss.elastic.co/t/log-pipeline/315714/3 "2022-11-04T22:32:10Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.