Which Filter to use


(Sjabiulla) #1

My log is something like below

{@timestamp} some text here key1=value1 key2=value2 key3=value3

I have tried json filter as below and got parsing errors. Which filter should I choose here ? I want to push all key value pairs and timestamp to variables in ElasticSearch.

json {
		source => "message"
	}

(Christian Dahlqvist) #2

As that log line is not JSON it is not surprising that the son filter does not work. I would recommend using a dissect or grok filter to parse the components of the string so that you end up with the final key-value list in a separate field. You can see how to go about doing this in this introductory blog post.

You can then apply a kv filter to this field to parse the key-value pairs out.


(Sjabiulla) #3

@Christian_Dahlqvist
I am using kv filter now as below. I have numbers as values in few of kv params, but in ElasticSearch they are being indexed as String's. How to index them as numbers !

      kv
		{ 
			source => "message" 
		} 

I have response time in log as below .
ResponseTimeMillis=454
But when it's indexed to ES, it's indexed as below.

"ResponseTimeMillis": "454"

I want it to be "ResponseTimeMillis": 454. Is it possible ?


(Christian Dahlqvist) #4

Yes, add a mutate filter to convert to a number after you have parsed the data.


(Sjabiulla) #5

Awesome. I'm now able to use mutate filter to convert string to a Integer.

Thank you