After switching from Filebeat 7.17 to Filebeat 8.1 and introducing type filestream into our configuration, it seems that Filebeat locks renamed logs forever, causing disk space issues.
We have a log4j based logs that roll over once reach 20MB, max 12 files.
Once the log is rolled, we noticed that there are hidden files created like below:
.nfs00000000000a0002000025a1
These files exist until Filebeat is restarted. I've just cleared 10GB of data just by restarting Filebeat.
A snippet (edited paths for security reasons) of our config:
filebeat.inputs:
- type: filestream
enabled: true
paths:
- /path/to/logs/integration/Integration.log
parsers:
- multiline:
type: pattern
pattern: '^\[[0-9]{2}-[0-9]{2}-[0-9]{2}[[:space:]]+[0-9]{2}:[0-9]{2}:[0-9]{2}'
negate: true
match: after
exclude_lines: ['Software', 'xml']
name: integ2_uat
tags: ["integ2_uat-logs"]
ignore_older: 24h
- type: filestream
enabled: true
paths:
- /path/to/logs/jboss_integration/server.log
parsers:
- multiline:
type: pattern
pattern: '^\[[0-9]{2}-[0-9]{2}-[0-9]{2}[[:space:]]+[0-9]{2}:[0-9]{2}:[0-9]{2}'
negate: true
match: after
exclude_lines: ['Software']
name: integ2_uat
tags: ["integ2_uat-logs_server"]
ignore_older: 24h
This behavior didn't occur before v8.1.
Which option would you advise to include in the configuration, avoiding any data loss?