Filebeat blocking generation of new log files by IIS

[gasparuben]

From time to time we are noting that filebeat some how is blocking generation of new log files by IIS process (w3wp.exe).
We follow a very standard ELK stack. Filebeat version 5.6.3 (64bits) runs on a WIndows 2008 R2 Enterprise server. It runs as a windows service, with following configuration for the collector:

...
- input_type: log
  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - E:\Logfiles\EWS\Ews_2*
  fields:  
    document_type: ews
  close_inactive: 5m
  close_renamed: true
  close_removed: true
  exclude_lines: ["^#"]
...

On the Windows system I see the open attributes as:

C:\..\SysinternalsSuite\handle.exe -u Ews_20180124

filebeat.exe       pid: 19800  type: File          NT AUTHORITY\SYSTEM        708: E:\Logfiles\EWS\Ews_20180124-142.LOG
filebeat.exe       pid: 19800  type: File          NT AUTHORITY\SYSTEM        7F4: E:\Logfiles\EWS\Ews_20180124-143.LOG
w3wp.exe           pid: 19580  type: File          NT AUTHORITY\SYSTEM       56B4: E:\Logfiles\EWS\Ews_20180124-143.LOG

C:\..\SysinternalsSuite\handle.exe | findstr /C:Ews_20180124-143.LOG
  7F4: File  (RWD)   E:\Logfiles\EWS\Ews_20180124-143.LOG
 56B4: File  (R--)   E:\Logfiles\EWS\Ews_20180124-143.LOG

Which is what I would expect.

To add more evidences, IIS creates the log file like :

|Desired Access:|Generic Write, Read Attributes|
|---|---|
|Disposition:|OpenIf|
|Options:|Synchronous IO Non-Alert, Non-Directory File, Open No Recall|
|Attributes:|n/a|
|ShareMode:|Read|
|AllocationSize:|0|
|OpenResult:|Created|

The symptoms reported are that from time to time we are missing the generation of new log files. So IIS fails to log new information on the old log file (rotation is done by size) while no new files are generated. Restarting the filebeat windows service seems like solving the issue.

Any hint?
Thank you!

[andrewkroh]

@gasparuben

Which is what I would expect.

Me too. Thanks for providing detailed debug information.

I don't have personal experience with IIS logging. So IIS is writing E:\Logfiles\EWS\Ews_20180124-143.LOG and then when it reaches a certain size it will start writing to E:\Logfiles\EWS\Ews_20180124-144.LOG (assuming it's in the same day)?

If so, I don't see how Filebeat would block the creation of the new file. Could IIS be blocked on a delete? Filebeat should allow for deleted as evidenced by the D is the share mode for the filebeat handle.