Close_removed does not work as expected on Windows


I use Filebeat 6.5.4 and I have a problem with close_removed attribute. I have an application with rotating logs hosted on Windows. When Elasticsearch is down for some time, Logstash persistent_queue is getting full and Filebeat's output to Logstash is blocked. Then, the harvester cannot release the files. When Logger tries to rotate file, the last rotated file is not removed and whole rotation is blocked, resulting in the active file's size growing above the limit. Such situation does not seem to happen with close_renamed file, which works as expected and closes the file once its renamed. One workaround would be to use close_timeout, but I would like to avoid a possibility of splitting multiline event.

Is it a close_removed bug that is locked or is there another option which can help me solving this problem?

Exception in Filebeat logs:

log/input.go:209	input state for D:\logs\app\app.log.5 was not removed: CreateFile D:\logs\app\app.log.5: Access is denied.

My filebeat config (I use close_timeout at the moment but I would like to avoid it)

- type: log
  enabled: true
  close_renamed: true
  close_removed: true
  close_inactive: 30s
  close_timeout: 2m
   application_type: application
   - D:\logs\app\app.log*
    pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
    negate: true
    match: after

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.