# Kv filtering problem

**URL:** <https://discuss.elastic.co/t/kv-filtering-problem/206277>\
**Category:** Logstash\
**Created:** [November 3, 2019, 8:05am UTC](https://discuss.elastic.co/t/kv-filtering-problem/206277 "2019-11-03T08:05:36Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![abu.sayeed](https://avatars.discourse-cdn.com/v4/letter/a/dec6dc/32.png) [@abu.sayeed](https://discuss.elastic.co/u/abu.sayeed)\
**Post date:** [November 3, 2019, 8:05am UTC](https://discuss.elastic.co/t/kv-filtering-problem/206277/1 "2019-11-03T08:05:36Z")

</div>

message" =\> **"SENT =\> Status : [SENT] | CID/GID : [20799461/ABC] | OBID : [08824] | SID : [02471] | StatusMsg : [null]",**

I wish the following fields  
Status =\> SENT,  
CID/GID =\> 20799461/ABC,  
OBID =\> 08824,  
SID =\> 02471,  
StatusMsg = null

My logstash grok patterns like:  
filter {  
kv {  
value\_split =\> ":"  
}  
}

But not working kv filtering. What is my wrong? Please help me to split that message.  
Thanks

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 3, 2019, 12:07pm UTC](https://discuss.elastic.co/t/kv-filtering-problem/206277/2 "2019-11-03T12:07:53Z")

</div>

That is not a strict KV format, so will require a combination of fields. First use adissect filter to separate everything but `SENT => ` in a single field, then replace `[` and `]` with empty strings unless you want these in the values. Then you should be able to use the KV filter on what remains with a `field_split` of `|` and a `value_split` of `:`.

---

<div class="post-metadata">

**Author:** ![abu.sayeed](https://avatars.discourse-cdn.com/v4/letter/a/dec6dc/32.png) [@abu.sayeed](https://discuss.elastic.co/u/abu.sayeed)\
**Post date:** [November 4, 2019, 6:01am UTC](https://discuss.elastic.co/t/kv-filtering-problem/206277/3 "2019-11-04T06:01:53Z")

</div>

Thank you for helping me.  
Yes, I have done according to your suggestions.  
Now my message like this.

message" =\> **" Status : [SENT] | CID/GID : [20799461/ABC] | OBID : [08824] | SID : [02471] | StatusMsg : [null]",**

I wish the following fields  
Status =\> SENT,  
CID/GID =\> 20799461/ABC,  
OBID =\> 08824,  
SID =\> 02471,  
StatusMsg = null

My logstash grok patterns like:  
filter {  
kv {  
field\_split =\> "|"  
value\_split =\> ":"  
}  
}

But not working kv filtering. Please help me to split that message.  
Thanks

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [November 4, 2019, 6:29am UTC](https://discuss.elastic.co/t/kv-filtering-problem/206277/4 "2019-11-04T06:29:18Z")

</div>

What does "not working" mean? Can you show what you are getting and the full config? Did you follow the other steps I described?

---

<div class="post-metadata">

**Author:** ![abu.sayeed](https://avatars.discourse-cdn.com/v4/letter/a/dec6dc/32.png) [@abu.sayeed](https://discuss.elastic.co/u/abu.sayeed)\
**Post date:** [November 4, 2019, 7:21am UTC](https://discuss.elastic.co/t/kv-filtering-problem/206277/5 "2019-11-04T07:21:33Z")

</div>

message =\> **" Status : [SENT] | CID/GID : [20799461/ABC] | OBID : [08824] | SID : [02471] | StatusMsg : [null]",**  
Class =\> "abc:567"  
host.name =\> "vm-1"  
@version =\> 1

filter {  
if [Class] == "abc:567" {  
kv {  
field\_split =\> "|"  
value\_split =\> ":"  
}  
mutate {  
remove\_field =\> ["message"]  
}  
}  
}

Logstash run successfully. And adissect filter working properly. But kv filter not working. Its show  
message =\> " Status : [SENT] | CID/GID : [20799461/ABC] | OBID : [08824] | SID : [02471] | StatusMsg : [null]",

Don't split any fields via kv filter. I try and try. But not find my fault sir.  
Thanks.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 4, 2019, 12:46pm UTC](https://discuss.elastic.co/t/kv-filtering-problem/206277/6 "2019-11-04T12:46:40Z")

</div>

If the [Class] field had the value "abc:567" at the point where that filter was processed then the [message] field would have been removed. It was not removed, so that suggests that the [Class] field is added later, and nothing in that filter section executes.

What does the complete configuration look like?

---

<div class="post-metadata">

**Author:** ![abu.sayeed](https://avatars.discourse-cdn.com/v4/letter/a/dec6dc/32.png) [@abu.sayeed](https://discuss.elastic.co/u/abu.sayeed)\
**Post date:** [November 5, 2019, 4:44am UTC](https://discuss.elastic.co/t/kv-filtering-problem/206277/7 "2019-11-05T04:44:25Z")

</div>

Log file like this:  
2019-11-05 10:08:25,452 : [INFO] http-5 [abc:571] SENT =\> Status : [nt-sent] | cId/gId : [20799461/ABC] | OBID : [08824] | SID : [18393567] | StatusMsg : [null]  
2019-11-05 10:08:25,453 : [INFO] http-5 [xyz:-1] Executing SP ACT\_nt with Action [UPDATE]

```
filter {
if [log][file][path] == "/home/jhon/jhon.log" {
grok {
  match => { "message" => "%{TIMESTAMP_ISO8601:time},%{DATA:ID} : \[%{DATA:loglevel}\] %{DATA:thread} \[%{DATA:class}\] %{DATA:bal} %{DATA:bal2} %{GREEDYDATA:message}" }
  overwrite => "message"
  }
}

else if [class] == "abc:571" {
kv {
  allow_duplicate_values => false
  include_brackets => true
  field_split => "|"
  value_split => ":"
  }
mutate {
    remove_field => ["message"]
  }
}

else if [Status] == "nt-sent" {
grok {
  match => { "cId/gId" => "%{DATA:cId}/%{DATA:gId}:" }
  }
mutate {
    remove_field => ["cId/gId"]
	}
  }
}

 			"thread" => "http-5",
 			 "class" => "abc:571",
	 "log.file.path" => "/home/jhon/jhon.log"
                "ID" => "796",
          "loglevel" => "INFO ",
 			  "time" => "2019-11-05 10:27:30",
        "@timestamp" => 2019-11-05T04:27:32.677Z,
           "message" => "Status : [nt-sent] | cId/gId : [20799461/ABC] | OBID : [08824] | SID : [18393567] | StatusMsg : [null]"

```

kv filtering not works.  
Thanks for helping.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [November 5, 2019, 3:19pm UTC](https://discuss.elastic.co/t/kv-filtering-problem/206277/8 "2019-11-05T15:19:33Z")

</div>

> [@abu.sayeed](#):
>
> kv filtering not works.

Correct. You configuration has

```
if [log][file][path] == "/home/jhon/jhon.log" {
    grok { ... }
}
else if [class] == "abc:571" {
    kv { ... }
}

```

If the grok is executed, which is what sets [class], then the else clause will never execute. Remove the word else.

Take a look at the trim\_key and trim\_value options on the kv filter.

---

<div class="post-metadata">

**Author:** ![abu.sayeed](https://avatars.discourse-cdn.com/v4/letter/a/dec6dc/32.png) [@abu.sayeed](https://discuss.elastic.co/u/abu.sayeed)\
**Post date:** [November 6, 2019, 11:30am UTC](https://discuss.elastic.co/t/kv-filtering-problem/206277/9 "2019-11-06T11:30:55Z")

</div>

Thanks a lots. Now all are ok

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 4, 2019, 11:31am UTC](https://discuss.elastic.co/t/kv-filtering-problem/206277/10 "2019-12-04T11:31:02Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.