# \[Basic Question\] - KV Filter avoid using a separator on a field

**URL:** <https://discuss.elastic.co/t/basic-question-kv-filter-avoid-using-a-separator-on-a-field/165680>\
**Category:** Logstash\
**Created:** [January 24, 2019, 11:47pm UTC](https://discuss.elastic.co/t/basic-question-kv-filter-avoid-using-a-separator-on-a-field/165680 "2019-01-24T23:47:50Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![Rh1n0](https://avatars.discourse-cdn.com/v4/letter/r/439d5e/32.png) [@Rh1n0](https://discuss.elastic.co/u/Rh1n0)\
**Post date:** [January 24, 2019, 11:47pm UTC](https://discuss.elastic.co/t/basic-question-kv-filter-avoid-using-a-separator-on-a-field/165680/1 "2019-01-24T23:47:50Z")

</div>

Guys hello, i'm very new to all the ELK stack. Right now i'm trying to parse some emailing logs that have an structure simillar to this:

```
    msgid=XXXXXX,
orig_msgid='<XXXXXX>',
relay=(), 
status='Email Deferred', 
received-email-over-tls='true', 
subject='=?UTF-8 Some Text that have a lot of =',
attachment(s)='file.doc',
number-attachment(s)='0'

```

And my KV Filter is this:

```
 kv {
 include_keys => [ "msgid","orig_msgid","relay","status","recieved-email-over- 
 tls","attachment(s)","number-attachment(s)" ]

```

Basically my problem is that i have a lot of events where the "subject" field has a lot of "=" characters inside the text and the words before the "=" are being considerated index by Kibana. My idea is to somehow use the grok filter to obtain the "subject" string, then somehow remove it from the "message" (or stored it somewhere else) and then use the KV filter to the message without the subject field/string.

Please i would appreciate some guidance with this.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 24, 2019, 11:58pm UTC](https://discuss.elastic.co/t/basic-question-kv-filter-avoid-using-a-separator-on-a-field/165680/2 "2019-01-24T23:58:33Z")

</div>

Perhaps something like this to remove the subject line

```
mutate { gsub => ["message", "subject='[^']+',", ""] }
```

---

<div class="post-metadata">

**Author:** ![Rh1n0](https://avatars.discourse-cdn.com/v4/letter/r/439d5e/32.png) [@Rh1n0](https://discuss.elastic.co/u/Rh1n0)\
**Post date:** [January 28, 2019, 7:55pm UTC](https://discuss.elastic.co/t/basic-question-kv-filter-avoid-using-a-separator-on-a-field/165680/3 "2019-01-28T19:55:41Z")

</div>

Thank you Badger i used your tip with some conditionals; basically if the key "subject" is in the message i extract the subject data with the grok filter , then i remove subject from message:

```
  if "subject" in [message] {

        grok {
                    match => { "message" => "subject=%{QUOTEDSTRING:subject}" }
        }

        mutate {
                    copy => { "message" => "filtered_message" }
        }

        mutate {
                gsub => ["filtered_message", "subject='[^']+',", ""]
        }

```

And then, i use the KV filter using filtered\_message as the input.

My problem now is that when the condition is met, filtered\_message appears as a field in Kibana and i want to avoid it to show on Kibana. I was thinking saving the field "filtered\_message" as a metadata field, but i can't figure it out the syntaxis.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [January 28, 2019, 9:25pm UTC](https://discuss.elastic.co/t/basic-question-kv-filter-avoid-using-a-separator-on-a-field/165680/4 "2019-01-28T21:25:15Z")

</div>

> [@Rh1n0](#):
>
> copy =\> { "message" =\> "filtered\_message" }

```
copy => { "message" => "[@metadata][filtered_message]" }

gsub => ["[@metadata][filtered_message]", "subject='[^']+',", ""]

```

---

<div class="post-metadata">

**Author:** ![Rh1n0](https://avatars.discourse-cdn.com/v4/letter/r/439d5e/32.png) [@Rh1n0](https://discuss.elastic.co/u/Rh1n0)\
**Post date:** [January 29, 2019, 9:48pm UTC](https://discuss.elastic.co/t/basic-question-kv-filter-avoid-using-a-separator-on-a-field/165680/5 "2019-01-29T21:48:52Z")

</div>

Thanks Badger i appreciate your help, that helped me to solve my problem!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 26, 2019, 9:54pm UTC](https://discuss.elastic.co/t/basic-question-kv-filter-avoid-using-a-separator-on-a-field/165680/6 "2019-02-26T21:54:35Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.