Filebeat 1.1.2 does not record correct inode values of rotated logs on exit

jstangroome · March 30, 2016, 3:30am

Hi,

I am seeing filebeat re-ship rotated log files when the filebeat service restarts.

I have created a basic configuration and bash script which starts filebeat, writes to a log file, rotates it a couple of times (with pauses in between to allow filebeat to notice) and then stops filebeat. Link at the end.

After filebeat stops, the registry has persisted the wrong inodes for the log files present on disk. As such, when filebeat starts again, an existing log file will appear to be a new file without an existing offset and its contents will be shipped a second time.

Regards,

Jason

Full repro script, conf, and output log available as a gist:

gist.github.com

https://gist.github.com/jstangroome/a1a70306cbd7a9420be15ec2bbe109a0

console.log

+ trap 'echo Exited with $?' EXIT
+ filebeat --version
+ grep --fixed-strings 1.1.2
filebeat version 1.1.2 (amd64)
+ sleep_duration_seconds=10
++ dirname /vagrant/filebeat-issue-repro.sh
+ conf_file=/vagrant/filebeat-issue-repro.yml
+ mkdir --parents /tmp/filebeat-issue-repro/
+ cd /tmp/filebeat-issue-repro/
+ rm --force ./input ./input.1 ./registry ./shipped

This file has been truncated. show original

filebeat-issue-repro.sh

#!/usr/bin/env bash
set -o errexit -o xtrace

trap "echo Exited with \$?" EXIT

filebeat --version | grep --fixed-strings 1.1.2

sleep_duration_seconds=10
conf_file="$(dirname "${0}")/filebeat-issue-repro.yml"

This file has been truncated. show original

filebeat-issue-repro.yml

filebeat:
  prospectors:
    -
      paths:
        - /tmp/filebeat-issue-repro/input*
      ignore_older: 24h
      max_backoff: 1s
      scan_frequency: 1s
  registry_file: /tmp/filebeat-issue-repro/registry
output:

This file has been truncated. show original

steffens · March 30, 2016, 11:24am

filebeat uses inode as reported by OS.

which filesystem are you using? Local or shared network?

jstangroome · March 30, 2016, 9:59pm

Hi Steffen,

We are experiencing the problem on a local ext4 file system on Ubuntu 14.04 running in AWS EC2.

My repro script was run on a local ext4 file systems on Ubuntu 14.04 running in a Virtualbox VM.

While filebeat remains running, it is correctly processing the files. The problem is that when it persists its state to the registry file, it is persisting the older inode values of the files before they were rotated, not the inodes of the files at those paths at the time filebeat exited.

Regards,

Jason

steffens · March 30, 2016, 10:57pm

Thanks for this very detailed test.

Looks a bug, with input.2 gone + not present in registry file there is no way filebeat can detect the rename. Please open an issue on github.

ruflin · April 12, 2016, 12:01pm

For reference: Here is a potential fix for the problem: https://github.com/elastic/beats/pull/1375