I'm using Filebeat 5.5.0 from https://artifacts.elastic.co/packages/5.x/apt stable main
in a Docker container running ubuntu:16.04 to ship logs from Docker containers to logstash.
When I start Filebeat, it correctly loads the config and starts prospectors. After a couple seconds, when Filebeat checks for changes in the specified file, it tries to do something with the running prospectors which seem to crash them. I get the following error logs:
reload.go:154: ERR Error creating module: Can only start a prospector when all related states are finished: {Source:/tmp/docker/containers/efc57384cecc856f9ec6f1118c88b74abe0aec058c256c49c65cd34a06d6242f/efc57384cecc856f9ec6f1118c88b74abe0aec058c256c49c65cd34a06d6242f-json.log Offset:1578 Finished:false Fileinfo:0xc4202c8270 FileStateOS:{Inode:13898764 Device:64513} Timestamp:2017-07-25 15:54:14.241126405 +0000 UTC TTL:-1ns}
filebeat.yml:
path.home: /usr/share/filebeat
path.conf: /usr/share/filebeat
path.data: /usr/share/filebeat/data
path.logs: /usr/share/filebeat/logs
filebeat.config.prospectors:
path: /usr/share/filebeat/generated/filebeat-generated.yml
reload.enabled: true
output:
logstash:
hosts: ["${LOGSTASH_HOST}:${LOGSTASH_PORT}"]
ssl.certificate: "/etc/ssl/myCert.pem"
ssl.key: "/etc/ssl/myKey.key"
index: filebeat
filebeat-generated:
- input_type: log
paths:
- /tmp/docker/containers/816c110d434fde19728d3d87e5657427e56c74d9d9e5363ea20e89eb4cce6867/816c110d434fde19728d3d87e5657427e56c74d9d9e5363ea20e89eb4cce6867-json.log
fields:
container_name: container1
- input_type: log
paths:
- /tmp/docker/containers/205ef75bd53120077eaf65054fafda46ee24ce2aa12c8b09c76b68b138fe4b12/205ef75bd53120077eaf65054fafda46ee24ce2aa12c8b09c76b68b138fe4b12-json.log
fields:
container_name: container2
- input_type: log
paths:
- /tmp/docker/containers/4eb6b312fe888e30713045e8a4fa6f09869f55fb2a5d7f10b1c7c10052453112/4eb6b312fe888e30713045e8a4fa6f09869f55fb2a5d7f10b1c7c10052453112-json.log
fields:
container_name: container3
Am I doing something wrong?
My issue seems similar to the following post which was supposedly fixed in FIlebeat 5.4.1
PS: I also tried using Filebeat 5.4.1 but got the same behaviour.