Filebeat harvests files again in combination with logrotate compress directive

[jori-be]

I'm using filebeat 8.11.3 on my systems and I notice that filebeat resubmits logs multiple times.

My application writes logs to the filesystem, logrotate rotates logs everyday.
We compress older log files also via logrotate.
After compressing the old logfile, we notice that the filesystem gives the inode from the old compressed file is used in the new logfile.
This confuses filebeat as it thinks the file got truncated as the inode was already known in the registry.
We see these logs in filebeat -> "File was truncated. Reading file from offset 0."

How can I prevent that filebeat gets confused here?
We prefer to keep the compression of the logs to safe disk space.

Filebeat config:

filebeat:
    inputs:
    -   close.on_state_change.inactive: 10m
        enabled: true
        id: some_id_log
        paths:
        - /data/log/app/appparsed/*/app-*.log*
        prospector.scanner.exclude_files:
        - \.gz$
        type: filestream

Logrotate:

/data/log/app/appparsed/*/app-*.log {
    missingok
    notifempty
    dateext
    dateformat _%Y-%m-%d_%H-%M-%S
    size 50M
    maxage 14
    rotate 240
    compress
    delaycompress
}

Any feedback or suggestions are welcome!

Thanks in advance!

[stefws (Steffen Winther Sørensen)]

@jori-be

delaycompress

Not sure about the logrotate parameter delaycompress, would this mean that an inode briefly would exist as rotated but without the .gz extention and thus can get pickup by filebeat's watcher... just a thought

[jori-be]

Hey @stefws ,

Thanks for the feedback.

The inode of the file doesn't change after delaycompress is executed. So filebeat should still know how far the file was processed before.

The problem actually starts when the files are compressed, then the compressed file gets a new inode and the inode of the original logfile is reused in the new active log file.

Best regards,

Jori

[stefws (Steffen Winther Sørensen)]

Inode reuse properly shouldn't happen shortly after, but dunno. Maybe try to force inodes out of registry relative fast with a short time in a clean_inactive parameter.

Or do as ChatGPT suggests

To ensure that Filebeat does not register an inode that points to a rotated compressed file, you need to make sure that Filebeat is aware of log rotation and handles it appropriately. Here's how you can adjust your configuration and log rotation scheme to achieve this:

Use log file renaming: Instead of compressing and rotating the log files in place, rename the log files before compression. This ensures that the inode being monitored by Filebeat always points to the active log file.
Modify log rotation settings: Adjust your log rotation settings to rename files before compression. You can achieve this by modifying your logrotate configuration as follows:
bash
Copy code
/data/log/app/appparsed/*/app-*.log {
    missingok
    notifempty
    dateext
    dateformat _%Y-%m-%d_%H-%M-%S
    size 50M
    maxage 14
    rotate 240
    # Renaming the file before compression
    postrotate
        mv /data/log/app/appparsed/*/app-*.log /data/log/app/appparsed/archive/
        gzip /data/log/app/appparsed/archive/app-*.log
    endscript
    delaycompress
}
This configuration will move the log files to an archive directory before compressing them. This way, Filebeat will continue to monitor the active log file and won't be affected by the rotation process.