Hi,
I use Filebeat 6.5.4 and I have a problem with close_removed attribute. I have an application with rotating logs hosted on Windows. When Elasticsearch is down for some time, Logstash persistent_queue is getting full and Filebeat's output to Logstash is blocked. Then, the harvester cannot release the files. When Logger tries to rotate file, the last rotated file is not removed and whole rotation is blocked, resulting in the active file's size growing above the limit. Such situation does not seem to happen with close_renamed file, which works as expected and closes the file once its renamed. One workaround would be to use close_timeout, but I would like to avoid a possibility of splitting multiline event.
Is it a close_removed bug that is locked or is there another option which can help me solving this problem?
Exception in Filebeat logs:
log/input.go:209 input state for D:\logs\app\app.log.5 was not removed: CreateFile D:\logs\app\app.log.5: Access is denied.
My filebeat config (I use close_timeout at the moment but I would like to avoid it)
- type: log
enabled: true
close_renamed: true
close_removed: true
close_inactive: 30s
close_timeout: 2m
fields:
application_type: application
paths:
- D:\logs\app\app.log*
multiline:
pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
negate: true
match: after