Custom message filter with logstash

[cuongnp (Phu Cuong Nguyen)]

Hi all,
I have a message log below:
<85>1 2021-05-11T09:25:02+07:00 172.19.16.241 CP-GW - Log [Fields@1.3.6.1.4.1.2620 Log delay="1620699902" src="172.19.9.18" dst="172.19.11.13" proto="6" UP_match_table="TABLE_START" ROW_START="0" match_id="35" layer_uuid="f5cec687-05e5-4573-b1dc-08119f24cbc9"
="Network" rule_uid="21e1b03b-5e20-4913-9d84-468164e67d8a" rule_name="" ROW_END="0" UP_match_table="TABLE_END" ProductName="VPN-1 & FireWall-1" svc="10051" sport_svc="64308" ProductFamily="Network" ].

I would like to define field for this message with output like this:
{
timestamp => 2021-05-11T09:25:02+07:00
src_ip => 172.19.9.18
dst_ip => 172.19.11.13
UP_match_table => TABLE_START
ROW_START => 0
match_id => 35
layer_uuid => f5cec687-05e5-4573-b1dc-08119f24cbc9
layer_name => Network
rule_uid => 21e1b03b-5e20-4913-9d84-468164e67d8a
rule_name => " "
ROW_END => 0
UP_match_table => TABLE_END
ProductName => VPN-1 & FireWall-1
svc => 10051
sport_svc => 64308
ProductFamily => Network
}
I read about logstash filter but I'm confused which filter I should use. Can you help me figure out

Thanks

[warkolm (Mark Walkom)]

You will want to use grok or dissect to do this for you.

[cuongnp (Phu Cuong Nguyen)]

Thanks Mark,

I tried with dissect filter, this is my configuration for logstash config

input { stdin { } }

filter {
dissect {
mapping => { 'message' => '%{} %{timestamp} %{} %{appname} %{} %{} [%{} %{} %{} %{service_id} %{ip_source} %{ip_dst} %{} %{up_match_table} %{row_start} %{match_id} %{layer_uuid} %{layer_name} %{rule_uid} %{rule_name} %{row_end} %{end_match_table} %{product_name} %{svc} %{sport_svc} %{product_family}]' }
}
}

output {
stdout { codec => rubydebug }
}

And I got output:

~

It created fields but value still contain field name, for example
"layer_name" => "layer_name="Network"" should be "layer_name" => "Network" instead

Thanks

[warkolm (Mark Walkom)]

Please don't post pictures of text, they are difficult to read, impossible to search and replicate (if it's code), and some people may not be even able to see them

[cuongnp (Phu Cuong Nguyen)]

Hi Mark, I repost the output, please have a look
<85>1 2021-05-11T11:13:43+07:00 172.19.16.243 CP-GW - Log [Fields@1.3.6.1.4.1.2620 Log delay="1620706423" service_id="http" src="172.19.15.14" dst="172.19.16.22" proto="6" UP_match_table="TABLE_START" ROW_START="0" match_id="39" layer_uuid="f5cec687-05e5-4573-b1dc-08119f24cbc9" layer_name="Network" rule_uid="eb1a5e3a-4f76-4a60-a3e1-9221b505a422" rule_name="TORtoFW" ROW_END="0" UP_match_table="TABLE_END" ProductName="VPN-1 & FireWall-1" svc="80" sport_svc="59304" ProductFamily="Network" ]
{
"layer_name" => "layer_name="Network"",
"sport_svc" => "FireWall-1"",
"appname" => "CP-GW",
"row_end" => "ROW_END="0"",
"ip_dst" => "dst="172.19.16.22"",
"@version" => "1",
"@timestamp" => 2021-05-11T04:35:47.407Z,
"product_family" => "svc="80" sport_svc="59304" ProductFamily="Network" ",
"layer_uuid" => "layer_uuid="f5cec687-05e5-4573-b1dc-08119f24cbc9"",
"product_name" => "ProductName="VPN-1",
"timestamp" => "2021-05-11T11:13:43+07:00",
"ip_source" => "src="172.19.15.14"",
"rule_uid" => "rule_uid="eb1a5e3a-4f76-4a60-a3e1-9221b505a422"",
"rule_name" => "rule_name="TORtoFW"",
"host" => "xplat-mon-01",
"row_start" => "ROW_START="0"",
"svc" => "&",
"up_match_table" => "UP_match_table="TABLE_START"",
"end_match_table" => "UP_match_table="TABLE_END"",
"match_id" => "match_id="39"",
"message" => "<85>1 2021-05-11T11:13:43+07:00 172.19.16.243 CP-GW - Log [Fields@1.3.6.1.4.1.2620 Log delay="1620706423" service_id="http" src="172.19.15.14" dst="172.19.16.22" proto="6" UP_match_table="TABLE_START" ROW_START="0" match_id="39" layer_uuid="f5cec687-05e5-4573-b1dc-08119f24cbc9" layer_name="Network" rule_uid="eb1a5e3a-4f76-4a60-a3e1-9221b505a422" rule_name="TORtoFW" ROW_END="0" UP_match_table="TABLE_END" ProductName="VPN-1 & FireWall-1" svc="80" sport_svc="59304" ProductFamily="Network" ]",
"service_id" => "service_id="http""

Thanks

[cuongnp (Phu Cuong Nguyen)]

It came out with "layer_name" => "layer_name="Network"", and I would like to ignore the specific text before "Network", so the result should be "layer_name" => "Network"

[Badger]

You might find it easier to use

    dissect { mapping => { 'message' => '%{} %{timestamp} %{} %{appname} %{} %{} [%{[@metadata][restOfLine]}]' } remove_field => [ "message" ] }
    kv { source => "[@metadata][restOfLine]" }