Filebeat prevents python rotating log from rotating files in windows os

Elastic Stack Beats
1

Environment:
win 10
python 2.7
filebeat 5.1.1

My application writes logs with Rotating log handler. When I start Filebeat I start getting the following exceptions from logging lib.

Traceback (most recent call last):
File "C:\Anaconda2\Lib\logging\handlers.py", line 77, in emit
self.doRollover()
File "C:\Anaconda2\Lib\logging\handlers.py", line 142, in doRollover
os.rename(self.baseFilename, dfn)

How can I resolve it?

2

What's the exception? Is there an error message?

3

sorry bad copy paste,

the error is:

WindowsError: [Error 183] Cannot create a file when that file already exists

4

Try close_removed.

1 Like
5

close_removed and close_renamed are enabled

my understanding from debugging the logger is:

the problem is when Python logger tries to rotate the file, he removes the old one and on that command I get exception, because Flebeat still holds the file open.

6

Oh, so the time between removing the old file and renaming is very short? So I guess it's not enough time for Filebeat to close the file. Maybe if you try using close_inactive the old file will closed by Filebeat before it's time for rotation.

7

posting my config:
it didn't work.
as you can see i aslo tried to add workers and pipelines

filebeat.prospectors:

  • input_type: log

    paths:

    • c:\Share\log*

close_renamed: true
close_removed: true
close_inactive: 30s

name:
steve_01
#----------------------------- Logstash output --------------------------------
output.logstash:

The Logstash hosts

hosts: ["host:port"]
worker: 2
pipelining: 2

8

The indentation is wrong. None of the close_x settings will be applied as written. These options are members of the prospector config and need to be indented at the same level as paths and input_type. Each prospector can use different settings therefore these are not global settings.

filebeat.prospectors:
- input_type: log
  paths:
  - my path
  close_inactive: 30s
1 Like
9

thanks for the help didn't see that.

10

This topic was automatically closed after 21 days. New replies are no longer allowed.