I am running the filebeat with below configuration. Filebeat always publishes all the messages from the log file whenever there is a new message written into it instead of reading the messages after the offset value. I see that offset is being written properly in registry file. Could some one help me to avoid publishing duplicate messages and read only new messages?
Hello @ash_coder this is a strange behavior, the only reason why Filebeat would start a file from the beginning is because the inode of the file on disk change. I am assuming that you are currently developing with the master branch.
To verify my theory:
Stop filebeat.
Get the inode of the file using ls -i myfile.log
Start filebeat
When filebeat send all the events from the file.
Get the inode of the file using ls -i myfile.log
I think the inode has changed.
Are you editing the log file with an editor and adding logs to it manually? If this is the case, some editors on saving will actually create a new file on disk, every new file has a new inode. When this happens Filebeat will read the file completely. If you want to keep the file you need to append to the file like this `echo "hello" >> myfile.log"
You have to append to the file to keep the same inode like
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.