Filebeat locks the file rotation for log4j causing the files to grow enormously


Sometimes the logs in my application are not rolled as they should be and they grow in size (the limit is 10mb)
I receive an exception if Filebeat logs:

Err: error setting up harvester: Harvester setup failed. Unexpected file opening error: Failed opening C:\application\logs\app.log.5: Error creating file 'C:\application\logs\app.log.5': Access is denied.
input state for C:\application\logs\app.log.2 was not removed: CreateFile C:\application\logs\app.log.2: Access is denied.

Then, Filebeat seems to keep a lock on the files on which the exception occured and they cannot be deleted/moved, so that logger keeps writing to the currently active file, which grows.

Are there tuning options or something to handle such situations?

I would like to make sure that if the log file is rotated before filebeat finishes reading it, I will not lose any data.

I use filebeat 6.4.2 with following input configuration :

    - type: log
      enabled: true
      close_renamed: true
       - C:\application\logs\app.log*
        pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
        negate: true
        match: after

I have the application which uses log4j for logging and it should rotate each log file whenever it reaches 10mb.

Best regards,

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.