Extract fields from log line


(Taoufiq Boukhcha) #1

Hello,

I need to extract KV fields from a java rest service log, the KV processor expect to have only key values, but this is what I have :

Oct 12 14:08:34 HOST_NAME SERVICE_NAME: 20171012T140834,059 INFO thread=thread_name cat=LoggerClassName [ keyOne="valueOne"  keyTwo="valueTwo"]  Request method='GET' path='/path/to/service' headers='accept-encoding:[gzip],connection:[Keep-Alive],content-type:[text/json; charset=utf-8]' payload='{\"name\":\"test with spaces and = this is\"}'   

I need to extract other non KV fields like timestamp and log level.

I want to get:

timestamp=20171012T140834,059
level=INFO
thread=thread_name
cat=LoggerClassName
method=GET
path=/path/to/service
headers=accept-encoding:[gzip],connection:[Keep-Alive],content-type:[text/json; charset=utf-8]
payload={\"name\":\"test with spaces and = this is\"}

message="Oct 12 14:08:34 HOST_NAME SERVICE_NAME: 20171012T140834,059 INFO thread=thread_name cat=LoggerClassName [ keyOne="valueOne"  keyTwo="valueTwo"]  Request method='GET' path='/path/to/service' headers='accept-encoding:[gzip],connection:[Keep-Alive],content-type:[text/json; charset=utf-8]' payload='{\"name\":\"test with spaces and = this is\"}'"

Any brilliant idea to do this with a minimum processors?

thanks


(David Pilato) #2

Probably a grok processor will do that.

But, I'm sure at some point you would like to convert timestamp to an actual date field...
So adding a date processor might also help.


(Taoufiq Boukhcha) #3

@dadoonet yes but in the part [ keyOne="valueOne" keyTwo="valueTwo"] I can have other keys (keyN="valueN") I don't know if we can do some dynamic things with grok (maybe I can extract whats between [...] in another field and use KV processor)

and some other logs (like error exception logs) does not have method, path, headers and payload is it possible to tell grok they're optional?


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.