Filebeat losing pointer and reading file from start

[paano]

Beats: 6.2.3

Have using filebeat for most of the our ELK solution to stream log files in to Elastic. IN one of the scenario, filebeat is reading the file from start.

scenario:
On a secure server, have rsync service shipping the log files to shared drive. Every time rsync is executed, delta logs is copied over to shared drive. And then the permission of the file is changed along with group.

Filebeat is polling the shared drive log file, and every time rsync is appending to the log file, Filebeat losses its pointer and starts reading the log file from top; thus causing duplicate entries into elastic.

Rsync command is as below

find /opt/appa/logs/ -mindepth 1 -mtime +7 -delete
rsync -ar /opt/appa/logs/   /shareddrv/appa/logs/testhost/ -o
chgrp appgrp /shareddrv/appa/logs/testhost/*
chmod -R 0770  /shareddrv/appa/logs/testhost/*

Is this a filebeat issue or the way rsync is appending the file?
Any body else seen this problem?

[pierhugues (Pier-Hugues Pellerin)]

Hello @paano, Filebeat internally uses the file inode to know if it's a new file or not.

I think by default the sync operation will by default creates new files and you have to use --append or --append-verify to append the data to the file instead of creating a new file.

[paano]

@pierhugues yes, i am using -a option to append the file.

rsync -ar /opt/appa/logs/ /shareddrv/appa/logs/testhost/ -o

Should be using the word --append

[pierhugues (Pier-Hugues Pellerin)]

Yes, from my rsync man page

        -a, --archive               archive mode; same as -rlptgoD (no -H)

I am pretty sure --append is different.