Extract fields from log line

[Taoufiq_BOUKHCHA]

Hello,

I need to extract KV fields from a java rest service log, the KV processor expect to have only key values, but this is what I have :

Oct 12 14:08:34 HOST_NAME SERVICE_NAME: 20171012T140834,059 INFO thread=thread_name cat=LoggerClassName [ keyOne="valueOne"  keyTwo="valueTwo"]  Request method='GET' path='/path/to/service' headers='accept-encoding:[gzip],connection:[Keep-Alive],content-type:[text/json; charset=utf-8]' payload='{\"name\":\"test with spaces and = this is\"}'   

I need to extract other non KV fields like timestamp and log level.

I want to get:

timestamp=20171012T140834,059
level=INFO
thread=thread_name
cat=LoggerClassName
method=GET
path=/path/to/service
headers=accept-encoding:[gzip],connection:[Keep-Alive],content-type:[text/json; charset=utf-8]
payload={\"name\":\"test with spaces and = this is\"}

message="Oct 12 14:08:34 HOST_NAME SERVICE_NAME: 20171012T140834,059 INFO thread=thread_name cat=LoggerClassName [ keyOne="valueOne"  keyTwo="valueTwo"]  Request method='GET' path='/path/to/service' headers='accept-encoding:[gzip],connection:[Keep-Alive],content-type:[text/json; charset=utf-8]' payload='{\"name\":\"test with spaces and = this is\"}'"

Any brilliant idea to do this with a minimum processors?

thanks

[dadoonet (David Pilato)]

Probably a grok processor will do that.

But, I'm sure at some point you would like to convert timestamp to an actual date field...
So adding a date processor might also help.

[Taoufiq_BOUKHCHA]

@dadoonet yes but in the part [ keyOne="valueOne" keyTwo="valueTwo"] I can have other keys (keyN="valueN") I don't know if we can do some dynamic things with grok (maybe I can extract whats between [...] in another field and use KV processor)

and some other logs (like error exception logs) does not have method, path, headers and payload is it possible to tell grok they're optional?