I am seeing filebeat re-ship rotated log files when the filebeat service restarts.
I have created a basic configuration and bash script which starts filebeat, writes to a log file, rotates it a couple of times (with pauses in between to allow filebeat to notice) and then stops filebeat. Link at the end.
After filebeat stops, the registry has persisted the wrong inodes for the log files present on disk. As such, when filebeat starts again, an existing log file will appear to be a new file without an existing offset and its contents will be shipped a second time.
Full repro script, conf, and output log available as a gist:
We are experiencing the problem on a local ext4 file system on Ubuntu 14.04 running in AWS EC2.
My repro script was run on a local ext4 file systems on Ubuntu 14.04 running in a Virtualbox VM.
While filebeat remains running, it is correctly processing the files. The problem is that when it persists its state to the registry file, it is persisting the older inode values of the files before they were rotated, not the inodes of the files at those paths at the time filebeat exited.