Filebeat close_inactive while file is still active

[hlamsc]

Hey there,

currently we have a problem with our filebeat version 6.3.2 on windows server 2016.
We have an application that logs massivly in one logfile. After the logfile size reached 20MB it gets rotated.
The problem:
Sometimes the rotation does not work because it seems like filebeat has still an active filehandle on it while on the same time saying that the logfile is inactive and the harvester is closed. Thus leading to very large logfiles and logs not beeing harvested. The only workaround so far is restarting the filebeat service.
The filebeat log contains serveral entries saying close_inactive (5m) is reached and the harvester is closed while the application log is still active and write logs so the inactivity should never be reached.
My guess is that the close is not really working, so the filehandle is still open preventing the application of rotating the logfile.

So this problem is leading me to two questions:

• Why does filebeat think that the logfile is inactive
• Why is close_inactive (probably) not closing the filehandle

Does anyone has similar problems or any ideas how to fix that?
Thanks!

Edit:
Config is kept simple:

> - type: log
>   enabled: true
>   paths:
> 
> fields:
>     log-type: xxx 
>     application_name: xxx
>     component: xxx
>     environment: Production
>     type: log
>   fields_under_root: true
> 
>   ### Multiline options
>  
>   multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2} [0-9]{2}:[0-9]{2}:[0-9]{2},[0-9]{3}'
>   multiline.negate: true
>   multiline.match: after

[faec (Fae Charlton)]

What may be happening with the messages you see is that close_inactive is referring to files that have already been rotated (Filebeat checks inactivity based on the file handle it has open, which will typically persist when a file is renamed, but will look "inactive" because new logs aren't written to the rotated file).

As for the rotation issues themselves, it depends on what method you use to rotate files, but there are known issues with some rotation methods on Windows -- you might want to look at the close_renamed option (see the docs here) to see if that fixes your problem.