Filebeat processing docker json-file logs

[evansj (Jon Evans)]

Hi

I'm trying to configure filebeat, in a docker container, to process docker container logs and send them to graylog. The docker log files are structured with a json message per line, like this:

{"log":"Starting contained process\n","stream":"stdout","time":"2017-10-19T14:16:33.648672118Z"}

I intend to run filebeat 6.0.0-rc1, so I can use the add_docker_metadata processor, but I can't even get it to work with docker.elastic.co/beats/filebeat:5.6.3 (without the processor).

The problem seems to be that the output json doesn't contain the "message" key. The messages arrive in graylog but the message is "null" and most of the other fields are missing.

For reference, this is my docker-compose.yml:

---
# Filebeat in docker
version: '2'
services:
  filebeat:
    image: docker.elastic.co/beats/filebeat:5.6.3
    container_name: filebeat
    user: root # Need to override user so we can access the log files, and docker.sock
    restart: always
    # Turn docker logging off, so this container won't try to log its own logs
    logging:
      driver: none
    # Override the "-e" which would have logged to stdout
    entrypoint:
      - filebeat
      - -c
      - filebeat.yml
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock"
      - "/mnt/data/docker/containers:/var/lib/docker/containers:ro"
      - "./logs:/usr/share/filebeat/logs"
      - "./config/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro"

and this is my filebeat.yml:

logging.level: debug
logging.to_files: true
logging.files:
  path: /usr/share/filebeat/logs
  name: debug.log
logging.selectors: ["*"]

filebeat.prospectors:
- type: log
  paths:
   - '/var/lib/docker/containers/*/*.log'
  json.message_key: log
  json.overwrite_keys: true
  json.keys_under_root: true
  json.add_error_key: true
output:
  logstash:
    hosts:
      - "my.logging.server:5044"

Here's an example of a published message, taken from debug.log:

2017-10-19T15:42:40Z DBG  Publish: {
  "@timestamp": "2017-10-19T15:42:39.712Z",
  "beat": {
    "hostname": "28e1239491bb",
    "name": "28e1239491bb",
    "version": "5.6.3"
  },
  "input_type": "log",
  "log": "2017-10-19 14:40:02,976 DEBUG org.springframework.beans.factory.support.DefaultListableBeanFactory : blah blah",
  "offset": 15345801,
  "source": "/var/lib/docker/containers/6e5bfd8a4b63194a51233f6440efe364561cbd01f43bd2151a9cef853d9e3269/6e5bfd8a4b63194a51233f6440efe364561cbd01f43bd2151a9cef853d9e3269-json.log",
  "stream": "stdout",
  "time": "2017-10-19T14:40:02.977120399Z",
  "type": "log"
}

Am I correct that there should be a "message" key in that json? I thought that json.message_key: log would add that (and remove the "log" key).

The messages in the graylog GUI contain the following keys / value examples:

facility: filebeat
file: /var/lib/docker/containers/.../...-json.log
input_type: log
message: null
name: 28e1239491bb
offset: 13249394
source: 28e1239491bb
timestamp: 2017-10-19T15:42:39.510Z
type: log

Please tell me where I'm going wrong.

[exekias (Carlos Pérez Aradros)]

Uhm, I think message_key should do that for you, will have a look to this, as it looks like a bug

[exekias (Carlos Pérez Aradros)]

I've been checking this further, after a second look at the docs: https://www.elastic.co/guide/en/beats/filebeat/master/configuration-filebeat-options.html#config-json.

The behavior is correct, message_key specifies what's the message field inside the json, so later in the chain processing (like multiline) applies to it. But it won't rename it to message. This would be an interesting feature though, so feel free to add a new request for enhancement in https://github.com/elastic/beats/issues