# Remove the beginning of message then apply kv filter
**URL:** https://discuss.elastic.co/t/remove-the-beginning-of-message-then-apply-kv-filter/205631
**Category:** Logstash
**Created:** [October 29, 2019, 10:00am UTC](https://discuss.elastic.co/t/remove-the-beginning-of-message-then-apply-kv-filter/205631 "2019-10-29T10:00:43Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@jameshanguyen](https://discuss.elastic.co/u/jameshanguyen)
#### Post date: [October 29, 2019, 10:00am UTC](https://discuss.elastic.co/t/remove-the-beginning-of-message-then-apply-kv-filter/205631/1 "2019-10-29T10:00:43Z")
Hi,
I have a message like this:
\<30\>device="SFW" date=2019-10-29 time=16:50:28 timezone="+07" device\_name="XG230" device\_id=C88877XHUG2BUU8 log\_id=010101600001 log\_type="Firewall" log\_component="Firewall Rule" log\_subtype="Allowed" status="Allow" priority=Information duration=30 fw\_rule\_id=106 policy\_type=1 user\_name="" user\_gp="" iap=0 ips\_policy\_id=0 appfilter\_policy\_id=8 application="DNS" application\_risk=1 application\_technology="Network Protocol" application\_category="Infrastructure" in\_interface="Port1" out\_interface="Port2" src\_mac=00:00:00:00:00:00 src\_ip=172.16.20.20 src\_country\_code=R1 dst\_ip=8.8.8.8 dst\_country\_code=USA protocol="UDP" src\_port=64647 dst\_port=53 sent\_pkts=1 recv\_pkts=1 sent\_bytes=90 recv\_bytes=106 tran\_src\_ip=105.39.39.100 tran\_src\_port=0 tran\_dst\_ip= tran\_dst\_port=0 srczonetype="LAN" srczone="LAN" dstzonetype="WAN" dstzone="WAN" dir\_disp="" connevent="Stop" connid="715948912" vconnid="" hb\_health="No Heartbeat" message="" appresolvedby="Signature" app\_is\_cloud=0
I would like to remove the beginning of the message (\<30\>device="SFW" date=2019-10-29 time=16:50:28 timezone="+07") and then kv filter the remaining.
How can I do that ?
Thanks.
---
### Author:  [@kolli\_dilip](https://discuss.elastic.co/u/kolli_dilip)
#### Post date: [October 29, 2019, 11:26am UTC](https://discuss.elastic.co/t/remove-the-beginning-of-message-then-apply-kv-filter/205631/2 "2019-10-29T11:26:32Z")
Hi @jameshanguyen
can you try a mutate filter with a regex as below
> mutate { gsub =\> ["message", "(\<.+?7" )(.+)", "\2"] }
or to be more generic use the below
> mutate { gsub =\> ["message", "(\<.+?=)(".+?")(.+?timezone=)(".+?")(.+)", "\5"] }
---
### Author:  [@jameshanguyen](https://discuss.elastic.co/u/jameshanguyen)
#### Post date: [October 29, 2019, 1:43pm UTC](https://discuss.elastic.co/t/remove-the-beginning-of-message-then-apply-kv-filter/205631/3 "2019-10-29T13:43:15Z")
Hi @kolli_dilip
thank you for your help.
How can I pass the new message to kv filter ?
Just like this ?
filter {
mutate { gsub =\> ["message", "(\<.+?7" )(.+)", "\2"] }
kv { }
}
---
### Author:  [@kolli\_dilip](https://discuss.elastic.co/u/kolli_dilip)
#### Post date: [October 29, 2019, 1:57pm UTC](https://discuss.elastic.co/t/remove-the-beginning-of-message-then-apply-kv-filter/205631/4 "2019-10-29T13:57:44Z")
yes, but configure your kv filter
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [October 29, 2019, 2:17pm UTC](https://discuss.elastic.co/t/remove-the-beginning-of-message-then-apply-kv-filter/205631/5 "2019-10-29T14:17:11Z")
There is no need to match the rest of the message or use capture groups. It would be simpler to do
```
mutate { gsub => ["message", "^<.+>", ""] }
kv { }
```
---
### Author:  [@jameshanguyen](https://discuss.elastic.co/u/jameshanguyen)
#### Post date: [October 29, 2019, 2:57pm UTC](https://discuss.elastic.co/t/remove-the-beginning-of-message-then-apply-kv-filter/205631/6 "2019-10-29T14:57:47Z")
hi @Badger
the elastic search can receive the log with
mutate { gsub =\> ["message", "^\<.+\>", ""] }
however, if I add
kv { }
then the elastic search doesn't receive the log anymore.
(If I use only kv { } without mutate, the elastic search doesn't receive the log either).
Does the kv { } take the result (new message) from mutate ? or it still uses the old message ?
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [October 29, 2019, 3:22pm UTC](https://discuss.elastic.co/t/remove-the-beginning-of-message-then-apply-kv-filter/205631/7 "2019-10-29T15:22:35Z")
With the kv filter the message field is not modified but additionals fields are added to the message
```
"iap" => "0",
"sent_pkts" => "1",
"dst_port" => "53",
"protocol" => "UDP",
"src_country_code" => "R1",
"device_name" => "XG230",
"src_mac" => "00:00:00:00:00:00",
```
etc. If this prevents the event reaching logstash it could be due to a mapping exception. Check the logstash and elasticsearch logs.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 26, 2019, 3:22pm UTC](https://discuss.elastic.co/t/remove-the-beginning-of-message-then-apply-kv-filter/205631/8 "2019-11-26T15:22:35Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.