Recently I've reported Filebeat beta1 resends random data upon every restart (registry file not updated properly?) where we didn't really reach a clear conclusion, but I was able to mitigate problem by removing clean_removed: true. But now it looks like there is another bug in a matter:
Repeatable scenario after upgrade to rc1:
- several boxes, all with the same configuration deployed by configuration management,
- filebeat is restarted at all of them by daily cron, at roughly the same time in the day,
- files in prospector paths meet the same criteria at all servers (similar volume, similar number of files, no files older than X (where is X is <10 ) hours)
- after restarts filebeat instance at RANDOM server might decide to rescan/resend entire data set from scratch. All other servers/filebeats behave properly, I see no key here.
- there is nothing in configuration that could trigger rescan. there is only:
clean_removed: false ignore_older: 25h
Few additional facts:
- dataset processed by every filebeat instance is rather huge; few dozens of GBs collected from few thousands of files,
- multiple prospectors in use, but not overlapping
- daily cron can call "service filebeat restart" twice in very short time, one after another,
- servers are rather busy,
- just by hoping to them I've noticed that sometimes registry file is not updated for several minutes, despite thousands of events being processed by said node,
Any help appreciated!