# Extract fields from log line

**URL:** <https://discuss.elastic.co/t/extract-fields-from-log-line/103902>\
**Category:** Elasticsearch\
**Created:** [October 13, 2017, 1:31pm UTC](https://discuss.elastic.co/t/extract-fields-from-log-line/103902 "2017-10-13T13:31:05Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Taoufiq\_BOUKHCHA](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/taoufiq_boukhcha/32/22993_2.png) [@Taoufiq\_BOUKHCHA](https://discuss.elastic.co/u/Taoufiq_BOUKHCHA)\
**Post date:** [October 13, 2017, 1:31pm UTC](https://discuss.elastic.co/t/extract-fields-from-log-line/103902/1 "2017-10-13T13:31:06Z")

</div>

Hello,

I need to extract KV fields from a java rest service log, the KV processor expect to have only key values, but this is what I have :

```auto
Oct 12 14:08:34 HOST_NAME SERVICE_NAME: 20171012T140834,059 INFO thread=thread_name cat=LoggerClassName [keyOne="valueOne" keyTwo="valueTwo"] Request method='GET' path='/path/to/service' headers='accept-encoding:[gzip],connection:[Keep-Alive],content-type:[text/json; charset=utf-8]' payload='{\"name\":\"test with spaces and = this is\"}'   

```

I need to extract other non KV fields like timestamp and log level.

I want to get:

```auto
timestamp=20171012T140834,059
level=INFO
thread=thread_name
cat=LoggerClassName
method=GET
path=/path/to/service
headers=accept-encoding:[gzip],connection:[Keep-Alive],content-type:[text/json; charset=utf-8]
payload={\"name\":\"test with spaces and = this is\"}

message="Oct 12 14:08:34 HOST_NAME SERVICE_NAME: 20171012T140834,059 INFO thread=thread_name cat=LoggerClassName [keyOne="valueOne" keyTwo="valueTwo"] Request method='GET' path='/path/to/service' headers='accept-encoding:[gzip],connection:[Keep-Alive],content-type:[text/json; charset=utf-8]' payload='{\"name\":\"test with spaces and = this is\"}'"

```

Any brilliant idea to do this with a minimum processors?

thanks

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [October 13, 2017, 1:44pm UTC](https://discuss.elastic.co/t/extract-fields-from-log-line/103902/2 "2017-10-13T13:44:45Z")

</div>

Probably a `grok` processor will do that.

But, I'm sure at some point you would like to convert `timestamp` to an actual date field...  
So adding a `date` processor might also help.

---

<div class="post-metadata">

**Author:** ![Taoufiq\_BOUKHCHA](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/taoufiq_boukhcha/32/22993_2.png) [@Taoufiq\_BOUKHCHA](https://discuss.elastic.co/u/Taoufiq_BOUKHCHA)\
**Post date:** [October 13, 2017, 1:55pm UTC](https://discuss.elastic.co/t/extract-fields-from-log-line/103902/3 "2017-10-13T13:55:40Z")

</div>

@dadoonet yes but in the part `[keyOne="valueOne" keyTwo="valueTwo"]` I can have other keys (keyN="valueN") I don't know if we can do some dynamic things with `grok` (maybe I can extract whats between `[...]` in another field and use KV processor)

and some other logs (like error exception logs) does not have `method`, `path`, `headers` and `payload` is it possible to tell grok they're optional?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 10, 2017, 1:55pm UTC](https://discuss.elastic.co/t/extract-fields-from-log-line/103902/4 "2017-11-10T13:55:48Z")

</div>

Closed on November 10, 2017, 1:55pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.