Custom message filter with logstash

Hi all,
I have a message log below:
<85>1 2021-05-11T09:25:02+07:00 172.19.16.241 CP-GW - Log [Fields@1.3.6.1.4.1.2620 Log delay="1620699902" src="172.19.9.18" dst="172.19.11.13" proto="6" UP_match_table="TABLE_START" ROW_START="0" match_id="35" layer_uuid="f5cec687-05e5-4573-b1dc-08119f24cbc9"
="Network" rule_uid="21e1b03b-5e20-4913-9d84-468164e67d8a" rule_name="" ROW_END="0" UP_match_table="TABLE_END" ProductName="VPN-1 & FireWall-1" svc="10051" sport_svc="64308" ProductFamily="Network" ].

I would like to define field for this message with output like this:
{
timestamp => 2021-05-11T09:25:02+07:00
src_ip => 172.19.9.18
dst_ip => 172.19.11.13
UP_match_table => TABLE_START
ROW_START => 0
match_id => 35
layer_uuid => f5cec687-05e5-4573-b1dc-08119f24cbc9
layer_name => Network
rule_uid => 21e1b03b-5e20-4913-9d84-468164e67d8a
rule_name => " "
ROW_END => 0
UP_match_table => TABLE_END
ProductName => VPN-1 & FireWall-1
svc => 10051
sport_svc => 64308
ProductFamily => Network
}
I read about logstash filter but I'm confused which filter I should use. Can you help me figure out

Thanks

You will want to use grok or dissect to do this for you.

Thanks Mark,

I tried with dissect filter, this is my configuration for logstash config

input { stdin { } }

filter {
dissect {
mapping => { 'message' => '%{} %{timestamp} %{} %{appname} %{} %{} [%{} %{} %{} %{service_id} %{ip_source} %{ip_dst} %{} %{up_match_table} %{row_start} %{match_id} %{layer_uuid} %{layer_name} %{rule_uid} %{rule_name} %{row_end} %{end_match_table} %{product_name} %{svc} %{sport_svc} %{product_family}]' }
}
}

output {
stdout { codec => rubydebug }
}

And I got output:

~

It created fields but value still contain field name, for example
"layer_name" => "layer_name="Network"" should be "layer_name" => "Network" instead

Thanks

Please don't post pictures of text, they are difficult to read, impossible to search and replicate (if it's code), and some people may not be even able to see them :slight_smile:

Hi Mark, I repost the output, please have a look
<85>1 2021-05-11T11:13:43+07:00 172.19.16.243 CP-GW - Log [Fields@1.3.6.1.4.1.2620 Log delay="1620706423" service_id="http" src="172.19.15.14" dst="172.19.16.22" proto="6" UP_match_table="TABLE_START" ROW_START="0" match_id="39" layer_uuid="f5cec687-05e5-4573-b1dc-08119f24cbc9" layer_name="Network" rule_uid="eb1a5e3a-4f76-4a60-a3e1-9221b505a422" rule_name="TORtoFW" ROW_END="0" UP_match_table="TABLE_END" ProductName="VPN-1 & FireWall-1" svc="80" sport_svc="59304" ProductFamily="Network" ]
{
"layer_name" => "layer_name="Network"",
"sport_svc" => "FireWall-1"",
"appname" => "CP-GW",
"row_end" => "ROW_END="0"",
"ip_dst" => "dst="172.19.16.22"",
"@version" => "1",
"@timestamp" => 2021-05-11T04:35:47.407Z,
"product_family" => "svc="80" sport_svc="59304" ProductFamily="Network" ",
"layer_uuid" => "layer_uuid="f5cec687-05e5-4573-b1dc-08119f24cbc9"",
"product_name" => "ProductName="VPN-1",
"timestamp" => "2021-05-11T11:13:43+07:00",
"ip_source" => "src="172.19.15.14"",
"rule_uid" => "rule_uid="eb1a5e3a-4f76-4a60-a3e1-9221b505a422"",
"rule_name" => "rule_name="TORtoFW"",
"host" => "xplat-mon-01",
"row_start" => "ROW_START="0"",
"svc" => "&",
"up_match_table" => "UP_match_table="TABLE_START"",
"end_match_table" => "UP_match_table="TABLE_END"",
"match_id" => "match_id="39"",
"message" => "<85>1 2021-05-11T11:13:43+07:00 172.19.16.243 CP-GW - Log [Fields@1.3.6.1.4.1.2620 Log delay="1620706423" service_id="http" src="172.19.15.14" dst="172.19.16.22" proto="6" UP_match_table="TABLE_START" ROW_START="0" match_id="39" layer_uuid="f5cec687-05e5-4573-b1dc-08119f24cbc9" layer_name="Network" rule_uid="eb1a5e3a-4f76-4a60-a3e1-9221b505a422" rule_name="TORtoFW" ROW_END="0" UP_match_table="TABLE_END" ProductName="VPN-1 & FireWall-1" svc="80" sport_svc="59304" ProductFamily="Network" ]",
"service_id" => "service_id="http""

Thanks

It came out with "layer_name" => "layer_name="Network"", and I would like to ignore the specific text before "Network", so the result should be "layer_name" => "Network"

You might find it easier to use

    dissect { mapping => { 'message' => '%{} %{timestamp} %{} %{appname} %{} %{} [%{[@metadata][restOfLine]}]' } remove_field => [ "message" ] }
    kv { source => "[@metadata][restOfLine]" }

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.