Filebeat skipping log lines in windows

[Skyy (Heather)]

Single Event Not Parsed Correctly

I just checked the inode, and it seems to be changing with each pull (tested manually), with the exception of a couple recent pulls, where it appears to keep going back and forth between two different inodes (2895/3098) However, even if I explicitly delete the file, then run the script to replace the file, I still get the same old inode of 2895. I'm wondering if clean_inactive could help here. Thanks, Cappy

So i have a similar issue outlined here and im pretty sure the problem is the same:
Our log writing process is custom and replaces the old log file entirely on update. The issue is when filebeat reads logs realtime and this occurs it is messing with the offset and clipping one or two lines. Im curious to know what filebeat uses on windows to identify files, is it also the inode?
I know its a problem that only occurs when reading realtime as it works flawlessly when ingesting older files.

Before I go through updating filebeat (and subsequent apps to keep them all playing nice), I have tried setting scan frequency to 5m. This works but not really... if the scan kicks off right as a new log file comes in i have the same issue. Is there a way to tell filebeat to ignore a file newer than 5 min (similar to ignore_older). That would fix my problem entirely.

[faec (Fae Charlton)]

This is a tricky case, especially on windows... when your log file swap happens, does it replace the log file with entirely new data, or is the beginning of the file unchanged? In the former case you'd want to re-ingest the whole file, in the latter it would be ok if it kept the previous offset. I'm not clear on what sequence is happening here that loses only one or two lines (if it's trying to resume in a totally different file then i'd expect it to lose more than that).

A setting you might find helpful is close_inactive, see the docs here. If your log files are updated atomically at known intervals then that should let you make sure filebeat isn't reading from them when the swap happens.