I'm using filebeat 8.11.3 on my systems and I notice that filebeat resubmits logs multiple times.
My application writes logs to the filesystem, logrotate rotates logs everyday.
We compress older log files also via logrotate.
After compressing the old logfile, we notice that the filesystem gives the inode from the old compressed file is used in the new logfile.
This confuses filebeat as it thinks the file got truncated as the inode was already known in the registry.
We see these logs in filebeat -> "File was truncated. Reading file from offset 0."
How can I prevent that filebeat gets confused here?
We prefer to keep the compression of the logs to safe disk space.
Not sure about the logrotate parameter delaycompress, would this mean that an inode briefly would exist as rotated but without the .gz extention and thus can get pickup by filebeat's watcher... just a thought
The inode of the file doesn't change after delaycompress is executed. So filebeat should still know how far the file was processed before.
The problem actually starts when the files are compressed, then the compressed file gets a new inode and the inode of the original logfile is reused in the new active log file.
Inode reuse properly shouldn't happen shortly after, but dunno. Maybe try to force inodes out of registry relative fast with a short time in a clean_inactive parameter.
Or do as ChatGPT suggests
To ensure that Filebeat does not register an inode that points to a rotated compressed file, you need to make sure that Filebeat is aware of log rotation and handles it appropriately. Here's how you can adjust your configuration and log rotation scheme to achieve this:
Use log file renaming: Instead of compressing and rotating the log files in place, rename the log files before compression. This ensures that the inode being monitored by Filebeat always points to the active log file.
Modify log rotation settings: Adjust your log rotation settings to rename files before compression. You can achieve this by modifying your logrotate configuration as follows:
bash
Copy code
/data/log/app/appparsed/*/app-*.log {
missingok
notifempty
dateext
dateformat _%Y-%m-%d_%H-%M-%S
size 50M
maxage 14
rotate 240
# Renaming the file before compression
postrotate
mv /data/log/app/appparsed/*/app-*.log /data/log/app/appparsed/archive/
gzip /data/log/app/appparsed/archive/app-*.log
endscript
delaycompress
}
This configuration will move the log files to an archive directory before compressing them. This way, Filebeat will continue to monitor the active log file and won't be affected by the rotation process.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.