[Basic Question] - KV Filter avoid using a separator on a field

Elastic Stack Logstash
1

Guys hello, i'm very new to all the ELK stack. Right now i'm trying to parse some emailing logs that have an structure simillar to this:

    msgid=XXXXXX,
orig_msgid='<XXXXXX>',
relay=(), 
status='Email Deferred', 
received-email-over-tls='true', 
subject='=?UTF-8 Some Text that have a lot of =',
attachment(s)='file.doc',
number-attachment(s)='0'

And my KV Filter is this:

 kv {
 include_keys => [  "msgid","orig_msgid","relay","status","recieved-email-over- 
 tls","attachment(s)","number-attachment(s)" ]

Basically my problem is that i have a lot of events where the "subject" field has a lot of "=" characters inside the text and the words before the "=" are being considerated index by Kibana. My idea is to somehow use the grok filter to obtain the "subject" string, then somehow remove it from the "message" (or stored it somewhere else) and then use the KV filter to the message without the subject field/string.

Please i would appreciate some guidance with this.

2

Perhaps something like this to remove the subject line

mutate { gsub => [ "message", "subject='[^']+',", ""] }
3

Thank you Badger i used your tip with some conditionals; basically if the key "subject" is in the message i extract the subject data with the grok filter , then i remove subject from message:

  if "subject" in [message] {

        grok {
                    match => { "message" => "subject=%{QUOTEDSTRING:subject}" }
        }

        mutate {
                    copy => { "message" =>  "filtered_message" }
        }

        mutate {
                gsub => [ "filtered_message", "subject='[^']+',", ""]
        }

And then, i use the KV filter using filtered_message as the input.

My problem now is that when the condition is met, filtered_message appears as a field in Kibana and i want to avoid it to show on Kibana. I was thinking saving the field "filtered_message" as a metadata field, but i can't figure it out the syntaxis.

4 
copy => { "message" =>  "[@metadata][filtered_message]" }

gsub => [ "[@metadata][filtered_message]", "subject='[^']+',", ""]
5

Thanks Badger i appreciate your help, that helped me to solve my problem!

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.