I´m new to ELK as well as to filebeat. Due to some restrictions we cannot install filebeat running on the source server(s), so we synchronize (rsync) all files regulary to the machine where filebeat is running.
After starting the sync, filebeat starts reading the whole file again - it´s not continuing at the latest offset.
How do I have to setup filebeat, so it remembers the last read position - and continues in processing only the new entries ?
Filebeat stores the last-read offset by default, but I suspect that the way rsync works (at least by default) breaks some of the assumptions it makes. In particular, Filebeat will think a file is new if:
the device/inode changed, even if the file name is the same
the file size is smaller than the current offset stored
I would recommend trying the --append or --append-verify option of rsync.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.