Filebeat locks the file rotation for log4j causing the files to grow enormously


(Captain Aye) #1

Hello,

Sometimes the logs in my application are not rolled as they should be and they grow in size (the limit is 10mb)
I receive an exception if Filebeat logs:

Err: error setting up harvester: Harvester setup failed. Unexpected file opening error: Failed opening C:\application\logs\app.log.5: Error creating file 'C:\application\logs\app.log.5': Access is denied.
input state for C:\application\logs\app.log.2 was not removed: CreateFile C:\application\logs\app.log.2: Access is denied.

Then, Filebeat seems to keep a lock on the files on which the exception occured and they cannot be deleted/moved, so that logger keeps writing to the currently active file, which grows.

Are there tuning options or something to handle such situations?

I would like to make sure that if the log file is rotated before filebeat finishes reading it, I will not lose any data.

I use filebeat 6.4.2 with following input configuration :

    - type: log
      enabled: true
      close_renamed: true
      paths:
       - C:\application\logs\app.log*
      multiline:
        pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
        negate: true
        match: after

I have the application which uses log4j for logging and it should rotate each log file whenever it reaches 10mb.

Best regards,


(system) closed #2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.