Using Grok with KV filter

Elastic Stack Logstash
1

Hi, I am pretty new to ELK stack. Currently I am trying to parse my application log using grok pattern. But since my logs are not structured i may have to write too many grok conditions, which will not scale well.
Sample log:
2019-02-25 10:22:27,832 LL="INFO" field1="field value" field2="field2val" MTHD="POST" O="ipadd" PAYLOAD="{"errorResponse":{"status":403,"message":"some message.","reason":"some reason"}}"
using KV filter I cant parse the above log, as the payload is not parsable using KV. (Correct me if I am wrong).
So I am planning to use both grok and KV, using grok I will get the time and get the payload, and remaining part will use KV filter?
Can you please help me here?
I am trying the below, but I am unable to use KV filter

grok {
  		match => { "message" => "%{TIMESTAMP_ISO8601:date}\s+%{GREEDYDATA:msgbody}"}
}

 grok { 	
	 	match => {
 		   "msgbody" => ["(?m).*%{GREEDYDATA:KV}\s+PAYLOAD=%{GREEDYDATA:PAYLOAD}"]
	  	}
   }
   
   kv {
       source => "KV"
       field_split => " "
       value_split => "="
       remove_char_key => "<>\[\],"
       remove_char_value => "<>\[\],"
       trim_key => "<>\[\],"
       trim_value => "<>\[\],"
       include_brackets => false

   }
2

Hi Satya,

Your KV is not using the field parsed by the grok filter. Here is the grok filter output:
image

Either rename the field in GROK to KV:

%{TIMESTAMP_ISO8601:date}\s+%{GREEDYDATA:KV}

Or you change the source field option in KV to msgbody:

kv {
       source => "msgbody"
       field_split => " "
       value_split => "="
   }
3

Thanks Nachiket for a quick reply.
But as I mentioned there is a PAYLOAD field which is a json. I want to save the payload as is in Elastic, if i parse the payload using KV, then it does not work..

4

Then use the json filter for the PAYLOAD field.

 json {
    source => "PAYLOAD"
    target => "payload_parsed"
  }
5

Change your second grok to

"msgbody" => ["%{GREEDYDATA:KV}\s+PAYLOAD=%{GREEDYDATA:PAYLOAD}"]
6

Awesome...this is what I was looking for, I have added .* before %{GREEDYDATA:KV}, thats a mistake.
Thanks a lot Badger

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.