How to configure prospectors when files are rsynced?

Elastic Stack Beats
1

Hi,

I´m new to ELK as well as to filebeat. Due to some restrictions we cannot install filebeat running on the source server(s), so we synchronize (rsync) all files regulary to the machine where filebeat is running.
After starting the sync, filebeat starts reading the whole file again - it´s not continuing at the latest offset.

How do I have to setup filebeat, so it remembers the last read position - and continues in processing only the new entries ?

Thanx for any hint,

Torsten

2

Filebeat stores the last-read offset by default, but I suspect that the way rsync works (at least by default) breaks some of the assumptions it makes. In particular, Filebeat will think a file is new if:

  • the device/inode changed, even if the file name is the same
  • the file size is smaller than the current offset stored

I would recommend trying the --append or --append-verify option of rsync.