Filebeat 1.1.2 does not record correct inode values of rotated logs on exit

Elastic Stack Beats
1

Hi,

I am seeing filebeat re-ship rotated log files when the filebeat service restarts.

I have created a basic configuration and bash script which starts filebeat, writes to a log file, rotates it a couple of times (with pauses in between to allow filebeat to notice) and then stops filebeat. Link at the end.

After filebeat stops, the registry has persisted the wrong inodes for the log files present on disk. As such, when filebeat starts again, an existing log file will appear to be a new file without an existing offset and its contents will be shipped a second time.

Regards,

Jason

Full repro script, conf, and output log available as a gist:

2

filebeat uses inode as reported by OS.

which filesystem are you using? Local or shared network?

3

Hi Steffen,

We are experiencing the problem on a local ext4 file system on Ubuntu 14.04 running in AWS EC2.

My repro script was run on a local ext4 file systems on Ubuntu 14.04 running in a Virtualbox VM.

While filebeat remains running, it is correctly processing the files. The problem is that when it persists its state to the registry file, it is persisting the older inode values of the files before they were rotated, not the inodes of the files at those paths at the time filebeat exited.

Regards,

Jason

4

Thanks for this very detailed test.

Looks a bug, with input.2 gone + not present in registry file there is no way filebeat can detect the rename. Please open an issue on github.

5

For reference: Here is a potential fix for the problem: https://github.com/elastic/beats/pull/1375