Kv filter has no support for this type of data {:type=>Hash, :value=>{"METHODNAME"=>

Elastic Stack Logstash
1

Can some please help me to parse the below log, I need the key-value pairs in DEBUGMESSAGE as my fields.

Here is my sample logs snippet:

{"thread":"default task-131","TimeStamp":"06/08/2021:00:58:02,848-08:00","level":"DEBUG","contextMap":"{HOST_NAME=192.168.2.193, SingleKey=1623214382841, TOKEN_ID=18665659791876978_NHV4K28W5i, day=9, logFileName=5487856, logger_key=202169online757054878561623214382529, month=6, state=online, store=7570, year=2021}","message":{"METHODNAME":"xxx","DEBUGMESSAGE":"{\"status\":\"true\",\"errorCode\":\"1001\",\"messages\":[{\"code\":\"1001\",\"description\":\"Request successfully processed\"}],\"requestID\":490004995,\"eligibleAmount\":1800.0,\"amountRequested\":1800.0,\"referenceID\":22278,\"productCode\":\"\",\"counterOffer\":false,\"includesFee\":false,\"responseTypeCode\":\"SUCCESS\",\"eligibilityReferenceID\":69795,\"denialCode\":\"\",\"passedCreditReview\":false,\"offerReferenceId\":79677,\"xxxReferenceID\":3011,\"correlationId\":\"7b07c144-30b9-4da3-bc58\",\"adverseActionDocumentId\":\"\"}","CLASSNAME":"xxxNewService"}}

Here is my Grok filter

input {
 beats {
   port => 5044
   type =>"%{[log_type]}"
      }
}

filter{
	
	if [fields][log_type] == "integrationlog" {
		grok{
			match => { "message" => "%{GREEDYDATA:json_data}"}
		} 	
		json{
			source => json_data
		}
		kv {
			source => "contextMap"
			value_split => "="
			field_split_pattern => ","
			remove_char_key => "\{\}"
			remove_char_value => "\{\}"
		}
		kv {
			source => "message"
			value_split => ":"
			field_split_pattern => ","
		}
		kv {
			source => "DEBUGMESSAGE"
			value_split => ":"
			field_split_pattern => ","
			remove_char_key => "\{\}"
			remove_char_value => "\{\}"
		}
		
	}
	
}
output {
	if [fields][log_type] == "integrationlog" {
		stdout { codec => rubydebug }
	}
	
}
2

I would replace the kv filters that try to parse message and DEBUGMESSAGE with

json { source => "[message][DEBUGMESSAGE]" }

If you need [message][METHODNAME] and [message][CLASSNAME] at the top level then use mutate+rename.

3

Many Thanks! It worked. :slightly_smiling_face: :slightly_smiling_face:

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.