Kv filtering problem

Elastic Stack Logstash
1

message" => "SENT => Status : [SENT] | CID/GID : [20799461/ABC] | OBID : [08824] | SID : [02471] | StatusMsg : [null]",

I wish the following fields
Status => SENT,
CID/GID => 20799461/ABC,
OBID => 08824,
SID => 02471,
StatusMsg = null

My logstash grok patterns like:
filter {
kv {
value_split => ":"
}
}

But not working kv filtering. What is my wrong? Please help me to split that message.
Thanks

2

That is not a strict KV format, so will require a combination of fields. First use adissect filter to separate everything but SENT => in a single field, then replace [ and ] with empty strings unless you want these in the values. Then you should be able to use the KV filter on what remains with a field_split of | and a value_split of :.

3

Thank you for helping me.
Yes, I have done according to your suggestions.
Now my message like this.

message" => " Status : [SENT] | CID/GID : [20799461/ABC] | OBID : [08824] | SID : [02471] | StatusMsg : [null]",

I wish the following fields
Status => SENT,
CID/GID => 20799461/ABC,
OBID => 08824,
SID => 02471,
StatusMsg = null

My logstash grok patterns like:
filter {
kv {
field_split => "|"
value_split => ":"
}
}

But not working kv filtering. Please help me to split that message.
Thanks

4

What does "not working" mean? Can you show what you are getting and the full config? Did you follow the other steps I described?

5

message => " Status : [SENT] | CID/GID : [20799461/ABC] | OBID : [08824] | SID : [02471] | StatusMsg : [null]",
Class => "abc:567"
host.name => "vm-1"
@version => 1

filter {
if [Class] == "abc:567" {
kv {
field_split => "|"
value_split => ":"
}
mutate {
remove_field => [ "message"]
}
}
}

Logstash run successfully. And adissect filter working properly. But kv filter not working. Its show
message => " Status : [SENT] | CID/GID : [20799461/ABC] | OBID : [08824] | SID : [02471] | StatusMsg : [null]",

Don't split any fields via kv filter. I try and try. But not find my fault sir.
Thanks.

6

If the [Class] field had the value "abc:567" at the point where that filter was processed then the [message] field would have been removed. It was not removed, so that suggests that the [Class] field is added later, and nothing in that filter section executes.

What does the complete configuration look like?

1 Like
7

Log file like this:
2019-11-05 10:08:25,452 : [INFO ] http-5 [abc:571] SENT => Status : [nt-sent] | cId/gId : [20799461/ABC] | OBID : [08824] | SID : [18393567] | StatusMsg : [null]
2019-11-05 10:08:25,453 : [INFO ] http-5 [xyz:-1] Executing SP ACT_nt with Action [UPDATE]

filter {
if [log][file][path] == "/home/jhon/jhon.log" {
grok {
  match => { "message" => "%{TIMESTAMP_ISO8601:time},%{DATA:ID} : \[%{DATA:loglevel}\] %{DATA:thread} \[%{DATA:class}\] %{DATA:bal} %{DATA:bal2} %{GREEDYDATA:message}" }
  overwrite => "message"
  }
}

else if [class] == "abc:571" {
kv {
  allow_duplicate_values => false
  include_brackets => true
  field_split => "|"
  value_split => ":"
  }
mutate {
    remove_field => [ "message"]
  }
}

else if [Status] == "nt-sent" {
grok {
  match => { "cId/gId" => "%{DATA:cId}/%{DATA:gId}:" }
  }
mutate {
    remove_field => [ "cId/gId"]
	}
  }
}




 			"thread" => "http-5",
 			 "class" => "abc:571",
	 "log.file.path" => "/home/jhon/jhon.log"
                "ID" => "796",
          "loglevel" => "INFO ",
 			  "time" => "2019-11-05 10:27:30",
        "@timestamp" => 2019-11-05T04:27:32.677Z,
           "message" => "Status : [nt-sent] | cId/gId : [20799461/ABC] | OBID : [08824] | SID : [18393567] |  StatusMsg : [null]"

kv filtering not works.
Thanks for helping.

8

Correct. You configuration has

if [log][file][path] == "/home/jhon/jhon.log" {
    grok { ... }
}
else if [class] == "abc:571" {
    kv { ... }
}

If the grok is executed, which is what sets [class], then the else clause will never execute. Remove the word else.

Take a look at the trim_key and trim_value options on the kv filter.

9

Thanks a lots. Now all are ok

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.