Hi
I'm trying to configure filebeat, in a docker container, to process docker container logs and send them to graylog. The docker log files are structured with a json message per line, like this:
{"log":"Starting contained process\n","stream":"stdout","time":"2017-10-19T14:16:33.648672118Z"}
I intend to run filebeat 6.0.0-rc1, so I can use the add_docker_metadata processor, but I can't even get it to work with docker.elastic.co/beats/filebeat:5.6.3 (without the processor).
The problem seems to be that the output json doesn't contain the "message" key. The messages arrive in graylog but the message is "null" and most of the other fields are missing.
For reference, this is my docker-compose.yml:
---
# Filebeat in docker
version: '2'
services:
filebeat:
image: docker.elastic.co/beats/filebeat:5.6.3
container_name: filebeat
user: root # Need to override user so we can access the log files, and docker.sock
restart: always
# Turn docker logging off, so this container won't try to log its own logs
logging:
driver: none
# Override the "-e" which would have logged to stdout
entrypoint:
- filebeat
- -c
- filebeat.yml
volumes:
- "/var/run/docker.sock:/var/run/docker.sock"
- "/mnt/data/docker/containers:/var/lib/docker/containers:ro"
- "./logs:/usr/share/filebeat/logs"
- "./config/filebeat.yml:/usr/share/filebeat/filebeat.yml:ro"
and this is my filebeat.yml:
logging.level: debug
logging.to_files: true
logging.files:
path: /usr/share/filebeat/logs
name: debug.log
logging.selectors: ["*"]
filebeat.prospectors:
- type: log
paths:
- '/var/lib/docker/containers/*/*.log'
json.message_key: log
json.overwrite_keys: true
json.keys_under_root: true
json.add_error_key: true
output:
logstash:
hosts:
- "my.logging.server:5044"
Here's an example of a published message, taken from debug.log:
2017-10-19T15:42:40Z DBG Publish: {
"@timestamp": "2017-10-19T15:42:39.712Z",
"beat": {
"hostname": "28e1239491bb",
"name": "28e1239491bb",
"version": "5.6.3"
},
"input_type": "log",
"log": "2017-10-19 14:40:02,976 DEBUG org.springframework.beans.factory.support.DefaultListableBeanFactory : blah blah",
"offset": 15345801,
"source": "/var/lib/docker/containers/6e5bfd8a4b63194a51233f6440efe364561cbd01f43bd2151a9cef853d9e3269/6e5bfd8a4b63194a51233f6440efe364561cbd01f43bd2151a9cef853d9e3269-json.log",
"stream": "stdout",
"time": "2017-10-19T14:40:02.977120399Z",
"type": "log"
}
Am I correct that there should be a "message"
key in that json? I thought that json.message_key: log
would add that (and remove the "log"
key).
The messages in the graylog GUI contain the following keys / value examples:
facility: filebeat
file: /var/lib/docker/containers/.../...-json.log
input_type: log
message: null
name: 28e1239491bb
offset: 13249394
source: 28e1239491bb
timestamp: 2017-10-19T15:42:39.510Z
type: log
Please tell me where I'm going wrong.