Filebeat skipping log lines in windows

Elastic Stack Beats
1

So i have a similar issue outlined here and im pretty sure the problem is the same:
Our log writing process is custom and replaces the old log file entirely on update. The issue is when filebeat reads logs realtime and this occurs it is messing with the offset and clipping one or two lines. Im curious to know what filebeat uses on windows to identify files, is it also the inode?
I know its a problem that only occurs when reading realtime as it works flawlessly when ingesting older files.

Before I go through updating filebeat (and subsequent apps to keep them all playing nice), I have tried setting scan frequency to 5m. This works but not really... if the scan kicks off right as a new log file comes in i have the same issue. Is there a way to tell filebeat to ignore a file newer than 5 min (similar to ignore_older). That would fix my problem entirely.

2

This is a tricky case, especially on windows... when your log file swap happens, does it replace the log file with entirely new data, or is the beginning of the file unchanged? In the former case you'd want to re-ingest the whole file, in the latter it would be ok if it kept the previous offset. I'm not clear on what sequence is happening here that loses only one or two lines (if it's trying to resume in a totally different file then i'd expect it to lose more than that).

A setting you might find helpful is close_inactive, see the docs here. If your log files are updated atomically at known intervals then that should let you make sure filebeat isn't reading from them when the swap happens.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.