# How can I expose new fields sourced from the log record?

**URL:** <https://discuss.elastic.co/t/how-can-i-expose-new-fields-sourced-from-the-log-record/62871>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [October 12, 2016, 10:30pm UTC](https://discuss.elastic.co/t/how-can-i-expose-new-fields-sourced-from-the-log-record/62871 "2016-10-12T22:30:49Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![kpam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kpam/32/12447_2.png) [@kpam](https://discuss.elastic.co/u/kpam)\
**Post date:** [October 12, 2016, 10:30pm UTC](https://discuss.elastic.co/t/how-can-i-expose-new-fields-sourced-from-the-log-record/62871/1 "2016-10-12T22:30:49Z")

</div>

Hi all,

I'm using a structured logging package ([GitHub - sirupsen/logrus: Structured, pluggable logging for Go.](https://github.com/Sirupsen/logrus)) and then shipping with Filebeat directly to our ES cluster. Unfortunately, I'm losing queryability on the fields I'm exposing since the entire log message gets wrapped in Filebeat's exported "message" field. Can someone point me in the right direction of being able to export these fields to be top-level?

For example, in I'm logging:

> log.WithFields(log.Fields{  
> "event": "click",  
> "topic": "video",  
> "key": "123",  
> }).Fatal("Failed to send event")

But on Kibana, we're reading in log records as such that have a field called 'message' which wraps up the logrus payload:  
But once this is propagated to Kibana, that log record is wrapped up:

> {"source": "/var/log/app.log",  
> message: "{"event":"click","topic":"video", "key"="123", "level"="fatal", "msg":"Failed to send event","time":"2015-08-12T18:47:07Z"}"  
> }

Is it possible to push fields such as 'key', 'event', and 'topic' to be top level so I can query?

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [October 12, 2016, 10:53pm UTC](https://discuss.elastic.co/t/how-can-i-expose-new-fields-sourced-from-the-log-record/62871/2 "2016-10-12T22:53:14Z")

</div>

It is possible to parse the JSON messages in Filebeat 5.x, but not in Filebeat 1.x. A [`json`](https://www.elastic.co/guide/en/beats/filebeat/5.0/configuration-filebeat-options.html#config-json) option can be specified in the configuration file.

If you are limited to using Filebeat 1.x, then you would need to Logstash to parse the JSON data from the `message` field. You would configure Filebeat -\> Logstash -\> Elasticsearch.

Filebeat 5.x configuration:

```
filebeat:
  prospectors:
    - paths:
        - /var/log/app.log
      json.message_key: msg
      json.keys_under_root: true
      json.add_error_key: true

output:
  console:
    pretty: true
```

---

<div class="post-metadata">

**Author:** ![kpam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kpam/32/12447_2.png) [@kpam](https://discuss.elastic.co/u/kpam)\
**Post date:** [October 13, 2016, 5:00pm UTC](https://discuss.elastic.co/t/how-can-i-expose-new-fields-sourced-from-the-log-record/62871/3 "2016-10-13T17:00:20Z")

</div>

Great, thank you @andrewkroh! I'll go ahead and give 5.x a shot.

My setup will be just Filebeat -\> Elasticsearch.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 2, 2016, 10:30pm UTC](https://discuss.elastic.co/t/how-can-i-expose-new-fields-sourced-from-the-log-record/62871/4 "2016-11-02T22:30:48Z")

</div>

This topic was automatically closed after 21 days. New replies are no longer allowed.