Reloading prospectors results in ERR in Filebeat 5.5.0

[dpaar]

I'm using Filebeat 5.5.0 from https://artifacts.elastic.co/packages/5.x/apt stable main in a Docker container running ubuntu:16.04 to ship logs from Docker containers to logstash.

When I start Filebeat, it correctly loads the config and starts prospectors. After a couple seconds, when Filebeat checks for changes in the specified file, it tries to do something with the running prospectors which seem to crash them. I get the following error logs:

reload.go:154: ERR Error creating module: Can only start a prospector when all related states are finished: {Source:/tmp/docker/containers/efc57384cecc856f9ec6f1118c88b74abe0aec058c256c49c65cd34a06d6242f/efc57384cecc856f9ec6f1118c88b74abe0aec058c256c49c65cd34a06d6242f-json.log Offset:1578 Finished:false Fileinfo:0xc4202c8270 FileStateOS:{Inode:13898764 Device:64513} Timestamp:2017-07-25 15:54:14.241126405 +0000 UTC TTL:-1ns}

filebeat.yml:

path.home: /usr/share/filebeat
path.conf: /usr/share/filebeat
path.data: /usr/share/filebeat/data
path.logs: /usr/share/filebeat/logs

filebeat.config.prospectors:
  path: /usr/share/filebeat/generated/filebeat-generated.yml
  reload.enabled: true

output:
  logstash:
    hosts: ["${LOGSTASH_HOST}:${LOGSTASH_PORT}"]
    ssl.certificate: "/etc/ssl/myCert.pem"
    ssl.key: "/etc/ssl/myKey.key"
    index: filebeat

filebeat-generated:

- input_type: log
  paths:
    - /tmp/docker/containers/816c110d434fde19728d3d87e5657427e56c74d9d9e5363ea20e89eb4cce6867/816c110d434fde19728d3d87e5657427e56c74d9d9e5363ea20e89eb4cce6867-json.log
  fields:
    container_name: container1
- input_type: log
  paths:
    - /tmp/docker/containers/205ef75bd53120077eaf65054fafda46ee24ce2aa12c8b09c76b68b138fe4b12/205ef75bd53120077eaf65054fafda46ee24ce2aa12c8b09c76b68b138fe4b12-json.log
  fields:
    container_name: container2
- input_type: log
  paths:
    - /tmp/docker/containers/4eb6b312fe888e30713045e8a4fa6f09869f55fb2a5d7f10b1c7c10052453112/4eb6b312fe888e30713045e8a4fa6f09869f55fb2a5d7f10b1c7c10052453112-json.log
  fields:
    container_name: container3

Am I doing something wrong?

My issue seems similar to the following post which was supposedly fixed in FIlebeat 5.4.1

Error when reloading prospectors for filebeat

I opened https://github.com/elastic/beats/pull/4128 with a potential fix. I still need to add tests and check it in some more details.

PS: I also tried using Filebeat 5.4.1 but got the same behaviour.

[ruflin]

Hi dpaar

Thanks for reporting this. I just had a look at our code and it could be that this error is actually logged even though it is prevented as the logging command seems to be at the wrong line.

Does it behave as expected or does it only log an error? Could you try to enable debug logging and check for the following line directly before the error line? Remove module from stoplist ... This makes sure running prospectors are not stopped.

Feel free to also just share the full log file in a gist.

[dpaar]

Thank you ruflin for your reply. Unfortunately I will not be able to test this until the week of August 21 as I'm going on vacation. I worked around this issue by restarting Filebeat everytime the config changes. However it would be nice to use prospector reloading in the future so I will set a reminder on this and get back to you.

From memory alone I think I did see the log line you mentioned as I was running Filebeat with the -v flag and I think the ERR actually prevented Filebeat prospectors from harvesting the Docker log files. But I will make sure once I get back.

[ruflin]

Thanks @dpaar Keep me posted. Feel free to also open a Github issue for this.

[dpaar]

I have been working on something else after my vacation but I will continue to work on our monitoring and logging solution in the coming weeks. I have not forgotten about you