How to overwrite/rewrite message field after kv and remove field filters?

Elastic Stack Logstash
1

Hi Experts,

I need to overwrite my message field once I am done with kv filter . My flow would be like

input{}
filter {
grok {    match=>["message",'CEF:%{CEFNUM:cef}[^\|\n]*\|%{CEFBAR:dvcvendor}\|%{CEFBAR:dvcproduct}\|%{CEFBAR:dvcversion}\|%{CEFBAR:alert}\|%{CEFDEV:alertmsg}[^\|\n]*\|%{CEFDEV:sev}[^\|\n]*\|'] }    
kv { 
   source => "message"
   field_split => "\s"
}
mutate {remove_field => [ "customerID","customerURI","modelConfidence","relevance"]}
grok {overwrite => ["message"]}
}

Here last grok filter is not working the way I want I still can see 4 fields in the message field .So idea is to overwrite final message field which does not have 4 fields in it .Please suggest how i can achieve this .

Thanks
VG

2

What do you want to overwrite the message field with? Using both grok and kv on the same field is a bit weird. If you can give an example log message it'll be easier to help.

3

So my original Message holds following, I have highlighted 4 fields which I want to remove.

Message: Mar 31 01:03:40 172.24.11.72 CEF:0|Check Point|VPN-1 & FireWall-1||drop|drop|High| eventId=23985829711 mrt=1459367155677 proto=TCP customerID=SO1jaHkUBABCK9DnjD5Ep6A customerURI=/XXX modelConfidence=asdkjdkjsdj relevance=blabla categorySignificance=/Informational/Warning categoryBehavior=/Access categoryDeviceGroup=/Firewall catdt=Firewall categoryOutcome=/Failure categoryObject=/Host/Application/Service modelConfidence=0 severity=5 relevance=10 assetCriticality=0 priority=High

What I want to see is a modified message field which should not have 4 fields(customerID,customerURI,modelConfidence and relevance) in it.

Message:Mar 31 01:03:40 172.24.11.72 CEF:0|Check Point|VPN-1 & FireWall-1||drop|drop|High| eventId=23985829711 mrt=1459367155677 proto=TCP categorySignificance=/Informational/Warning categoryBehavior=/Access categoryDeviceGroup=/Firewall catdt=Firewall categoryOutcome=/Failure categoryObject=/Host/Application/Service modelConfidence=0 severity=5 relevance=10 assetCriticality=0 priority=High

4

Perhaps you can just use the mutate filter's gsub option?