Filebeat prevents python rotating log from rotating files in windows os

(Dima Shechtman) #1

win 10
python 2.7
filebeat 5.1.1

My application writes logs with Rotating log handler. When I start Filebeat I start getting the following exceptions from logging lib.

Traceback (most recent call last):
File "C:\Anaconda2\Lib\logging\", line 77, in emit
File "C:\Anaconda2\Lib\logging\", line 142, in doRollover
os.rename(self.baseFilename, dfn)

How can I resolve it?

(Andrew Kroh) #2

What's the exception? Is there an error message?

(Dima Shechtman) #3

sorry bad copy paste,

the error is:

WindowsError: [Error 183] Cannot create a file when that file already exists

(Andrew Kroh) #4

Try close_removed.

(Dima Shechtman) #5

close_removed and close_renamed are enabled

my understanding from debugging the logger is:

the problem is when Python logger tries to rotate the file, he removes the old one and on that command I get exception, because Flebeat still holds the file open.

(Andrew Kroh) #6

Oh, so the time between removing the old file and renaming is very short? So I guess it's not enough time for Filebeat to close the file. Maybe if you try using close_inactive the old file will closed by Filebeat before it's time for rotation.

(Dima Shechtman) #7

posting my config:
it didn't work.
as you can see i aslo tried to add workers and pipelines


  • input_type: log


    • c:\Share\log*

close_renamed: true
close_removed: true
close_inactive: 30s

#----------------------------- Logstash output --------------------------------

The Logstash hosts

hosts: ["host:port"]
worker: 2
pipelining: 2

(Andrew Kroh) #8

The indentation is wrong. None of the close_x settings will be applied as written. These options are members of the prospector config and need to be indented at the same level as paths and input_type. Each prospector can use different settings therefore these are not global settings.

- input_type: log
  - my path
  close_inactive: 30s

(Dima Shechtman) #9

thanks for the help didn't see that.

(system) #10

This topic was automatically closed after 21 days. New replies are no longer allowed.