This is the 2nd time this has happened now. I have a prospector setup to read /var/log/myapp/*.log and it brings up about 5 or 6 harvesters as intended. Things run smooth for a while but what seems to be randomly it just stops harvesting one of the files. The file is continually being written to and rotates the same time as all the others. The other prospectors are running just fine and if I restart filebeat it picks the missing file backup and starts shipping again. I don't see any errors in the mybeat logs either.
filebeat: registry_file: /var/lib/filebeat/registry config_dir: /etc/filebeat output: logstash: enabled: true hosts: ["redacted.com:5044"]
filebeat: prospectors: - paths: - /var/log/myapp/*.log input_type: log ignore_older: 10m document_type: core
Here's some log files that might help. Notice the timestamp. After this no info for that particular log is generated (starting harvester, stopping, etc)
2015-12-17T07:17:55Z ERR File reading error. Stopping harvester. Error: Stop harvesting as file is older then ignore_older: /var/log/myapp/mylog.log; Last change was: 10m5.015760004s 2015-12-17T07:53:45Z ERR File reading error. Stopping harvester. Error: Stop harvesting as file is older then ignore_older: /var/log/myapp/mylog.log; Last change was: 10m5.016134511s
/var/log/myapp/mylog.log: Again looking at the timestamps it should be picking this up. I redacted everything except the timestamp and loglevel just to show that the file is being written to.
2015-12-17 07:48:57.731427 [INFO] 2015-12-17 07:56:17.676947 [INFO] 2015-12-17 07:56:17.730149 [INFO] 2015-12-17 07:56:17.733369 [INFO] 2015-12-17 07:56:17.734734 [INFO] 2015-12-17 07:56:17.734842 [INFO] 2015-12-17 07:56:17.739996 [INFO]
Any ideas on what else I could look at? We are very close to pushing filebeat into production but this 'unknown' is holding it back. Thanks.