Have using filebeat for most of the our ELK solution to stream log files in to Elastic. IN one of the scenario, filebeat is reading the file from start.
On a secure server, have
rsync service shipping the log files to shared drive. Every time
rsync is executed,
delta logs is copied over to shared drive. And then the permission of the file is changed along with group.
Filebeat is polling the shared drive log file, and every time
rsync is appending to the log file, Filebeat losses its pointer and starts reading the log file from top; thus causing duplicate entries into elastic.
Rsync command is as below
find /opt/appa/logs/ -mindepth 1 -mtime +7 -delete rsync -ar /opt/appa/logs/ /shareddrv/appa/logs/testhost/ -o chgrp appgrp /shareddrv/appa/logs/testhost/* chmod -R 0770 /shareddrv/appa/logs/testhost/*
Is this a
filebeat issue or the way
rsync is appending the file?
Any body else seen this problem?