We are looking to use autodiscovery with Filebeat in Kubernetes to make it possible to configure Filebeat on a pod by pod basis, as it is intended.
However, when enabling we are swamped with Error creating runner from config: Can only start an input when all related states are finished errors.
Full example:
2019-03-18T23:33:50.024Z ERROR [autodiscover] cfgfile/list.go:96 Error creating runner from config: Can only start an input when all related states are finished: {Id:5028632-2049 Finished:false Fileinfo:0xc4230b41a0 Source:/var/lib/docker/containers/b374368c7d0f435be211c4d6f0e5e86a6e130a284648b069b491f3cd2acb6f74/b374368c7d0f435be211c4d6f0e5e86a6e130a284648b069b491f3cd2acb6f74-json.log Offset:379130 Timestamp:2019-03-18 23:27:58.030604195 +0000 UTC m=+620.858811210 TTL:-1ns Type:docker Meta:map[] FileStateOS:5028632-2049}
The config we are running:
filebeat.autodiscover:
providers:
- type: kubernetes
hints.enabled: true
output.file:
enabled: false
output.logstash:
enabled: true
hosts: ["logstash:5044"]
Filebeat version is 6.6.2.
When running this configuration Filebeat seems to perform poorly with the logs that actually work taking sometime before appearing in Kibana. Some logs don't seem to write at all.
When reverting to a regular filebeat.input configuration and removing autodiscovery Filebeat works again as normal and this error ceases.
Understand this error might be due to multiple inputs trying to read the same file but as you can see the in the config we only run the autodiscovery config.
Any suggestions?
Thanks!
Tom