Remove the beginning of message then apply kv filter

[jameshanguyen]

Hi,
I have a message like this:

<30>device="SFW" date=2019-10-29 time=16:50:28 timezone="+07" device_name="XG230" device_id=C88877XHUG2BUU8 log_id=010101600001 log_type="Firewall" log_component="Firewall Rule" log_subtype="Allowed" status="Allow" priority=Information duration=30 fw_rule_id=106 policy_type=1 user_name="" user_gp="" iap=0 ips_policy_id=0 appfilter_policy_id=8 application="DNS" application_risk=1 application_technology="Network Protocol" application_category="Infrastructure" in_interface="Port1" out_interface="Port2" src_mac=00:00:00:00:00:00 src_ip=172.16.20.20 src_country_code=R1 dst_ip=8.8.8.8 dst_country_code=USA protocol="UDP" src_port=64647 dst_port=53 sent_pkts=1 recv_pkts=1 sent_bytes=90 recv_bytes=106 tran_src_ip=105.39.39.100 tran_src_port=0 tran_dst_ip= tran_dst_port=0 srczonetype="LAN" srczone="LAN" dstzonetype="WAN" dstzone="WAN" dir_disp="" connevent="Stop" connid="715948912" vconnid="" hb_health="No Heartbeat" message="" appresolvedby="Signature" app_is_cloud=0

I would like to remove the beginning of the message (<30>device="SFW" date=2019-10-29 time=16:50:28 timezone="+07") and then kv filter the remaining.
How can I do that ?
Thanks.

[kolli_dilip]

Hi @jameshanguyen

can you try a mutate filter with a regex as below

mutate { gsub => [ "message", "(<.+?7" )(.+)", "\2" ] }

or to be more generic use the below

mutate { gsub => [ "message", "(<.+?=)(".+?")(.+?timezone=)(".+?")(.+)", "\5" ] }

[jameshanguyen]

Hi @kolli_dilip
thank you for your help.
How can I pass the new message to kv filter ?
Just like this ?

filter {
mutate { gsub => [ "message", "(<.+?7" )(.+)", "\2" ] }
kv { }
}

[kolli_dilip]

yes, but configure your kv filter

[Badger]

There is no need to match the rest of the message or use capture groups. It would be simpler to do

    mutate { gsub => [ "message", "^<.+>", "" ] }
    kv { }

[jameshanguyen]

hi @Badger
the elastic search can receive the log with
mutate { gsub => [ "message", "^<.+>", "" ] }

however, if I add
kv { }
then the elastic search doesn't receive the log anymore.

(If I use only kv { } without mutate, the elastic search doesn't receive the log either).

Does the kv { } take the result (new message) from mutate ? or it still uses the old message ?

[Badger]

With the kv filter the message field is not modified but additionals fields are added to the message

                   "iap" => "0",
             "sent_pkts" => "1",
              "dst_port" => "53",
              "protocol" => "UDP",
      "src_country_code" => "R1",
           "device_name" => "XG230",
               "src_mac" => "00:00:00:00:00:00",

etc. If this prevents the event reaching logstash it could be due to a mapping exception. Check the logstash and elasticsearch logs.