# Filebeat processing docker json-file logs

**URL:** <https://discuss.elastic.co/t/filebeat-processing-docker-json-file-logs/104590>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [October 19, 2017, 4:14pm UTC](https://discuss.elastic.co/t/filebeat-processing-docker-json-file-logs/104590 "2017-10-19T16:14:44Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![evansj](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/evansj/32/23197_2.png) [@evansj](https://discuss.elastic.co/u/evansj)\
**Post date:** [October 19, 2017, 4:14pm UTC](https://discuss.elastic.co/t/filebeat-processing-docker-json-file-logs/104590/1 "2017-10-19T16:14:44Z")

</div>

Hi

I'm trying to configure filebeat, in a docker container, to process docker container logs and send them to graylog. The docker log files are structured with a json message per line, like this:

```
{"log":"Starting contained process\n","stream":"stdout","time":"2017-10-19T14:16:33.648672118Z"}

```

I intend to run filebeat 6.0.0-rc1, so I can use the add\_docker\_metadata processor, but I can't even get it to work with [docker.elastic.co/beats/filebeat:5.6.3](http://docker.elastic.co/beats/filebeat:5.6.3) (without the processor).

The problem seems to be that the output json doesn't contain the "message" key. The messages arrive in graylog but the message is "null" and most of the other fields are missing.

For reference, this is my docker-compose.yml:

```auto
---
# Filebeat in docker
version: '2'
services:
  filebeat:
    image: docker.elastic.co/beats/filebeat:5.6.3
    container_name: filebeat
    user: root # Need to override user so we can access the log files, and docker.sock
    restart: always
    # Turn docker logging off, so this container won't try to log its own logs
    logging:
      driver: none
    # Override the "-e" which would have logged to stdout
    entrypoint:
      - filebeat
      - -c
      - filebeat.yml
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"
      - "/mnt/data/docker/containers:/var/lib/docker/containers:ro"
      - "./logs:/usr/share/filebeat/logs"
      - "./config/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro"

```

and this is my filebeat.yml:

```auto
logging.level: debug
logging.to_files: true
logging.files:
  path: /usr/share/filebeat/logs
  name: debug.log
logging.selectors: ["*"]

filebeat.prospectors:
- type: log
  paths:
   - '/var/lib/docker/containers/*/*.log'
  json.message_key: log
  json.overwrite_keys: true
  json.keys_under_root: true
  json.add_error_key: true
output:
  logstash:
    hosts:
      - "my.logging.server:5044"

```

Here's an example of a published message, taken from debug.log:

```auto
2017-10-19T15:42:40Z DBG Publish: {
  "@timestamp": "2017-10-19T15:42:39.712Z",
  "beat": {
    "hostname": "28e1239491bb",
    "name": "28e1239491bb",
    "version": "5.6.3"
  },
  "input_type": "log",
  "log": "2017-10-19 14:40:02,976 DEBUG org.springframework.beans.factory.support.DefaultListableBeanFactory : blah blah",
  "offset": 15345801,
  "source": "/var/lib/docker/containers/6e5bfd8a4b63194a51233f6440efe364561cbd01f43bd2151a9cef853d9e3269/6e5bfd8a4b63194a51233f6440efe364561cbd01f43bd2151a9cef853d9e3269-json.log",
  "stream": "stdout",
  "time": "2017-10-19T14:40:02.977120399Z",
  "type": "log"
}

```

Am I correct that there should be a `"message"` key in that json? I thought that `json.message_key: log` would add that (and remove the `"log"` key).

The messages in the graylog GUI contain the following keys / value examples:

```auto
facility: filebeat
file: /var/lib/docker/containers/.../...-json.log
input_type: log
message: null
name: 28e1239491bb
offset: 13249394
source: 28e1239491bb
timestamp: 2017-10-19T15:42:39.510Z
type: log

```

Please tell me where I'm going wrong. 🙂

---

<div class="post-metadata">

**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\
**Post date:** [October 23, 2017, 8:52am UTC](https://discuss.elastic.co/t/filebeat-processing-docker-json-file-logs/104590/2 "2017-10-23T08:52:29Z")

</div>

Uhm, I think `message_key` should do that for you, will have a look to this, as it looks like a bug

---

<div class="post-metadata">

**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\
**Post date:** [October 24, 2017, 11:32am UTC](https://discuss.elastic.co/t/filebeat-processing-docker-json-file-logs/104590/3 "2017-10-24T11:32:32Z")

</div>

I've been checking this further, after a second look at the docs: [https://www.elastic.co/guide/en/beats/filebeat/master/configuration-filebeat-options.html#config-json](https://www.elastic.co/guide/en/beats/filebeat/master/configuration-filebeat-options.html#config-json).

The behavior is correct, message\_key specifies what's the message field inside the json, so later in the chain processing (like multiline) applies to it. But it won't rename it to `message`. This would be an interesting feature though, so feel free to add a new request for enhancement in [https://github.com/elastic/beats/issues](https://github.com/elastic/beats/issues)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 21, 2017, 11:32am UTC](https://discuss.elastic.co/t/filebeat-processing-docker-json-file-logs/104590/4 "2017-11-21T11:32:38Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.