I have a set of logs that are being rotated and then removed from a directory, but filebeat still has the harvester opened on said file and it is not letting go of it. In my directory you see the number of logs that are deleted, but they are still taking up the space.
filebeat 2209 user 4r REG 253,3 4147407173 77988311 /some/local/dummy/export/path/ratedata_2018-03-20T04-51-37.821.csv (deleted)
I have read through the documentation and am aware that filebeat will not close the harvester until it has finished reading the data. Once i restart the agent the file system goes down. I have used some of these settings on our config:
- input_type: log paths: - /my_path/* #Excludes .csv files exclude_files: ['\.csv'] #Closes/Releases if file is not written to in 5 minutes. close_inactive: 5m scan_frequency: 10s tail_files: true enabled: true fields_under_root: true fields: service: my_service
Why is filebeat still holding onto the file?
Are there any other "best practice" settings i should be using?
Thanks for the help.