Filebeat not releasing file after it's been deleted

[cchooks2 (Steven)]

Hello,

I have a set of logs that are being rotated and then removed from a directory, but filebeat still has the harvester opened on said file and it is not letting go of it. In my directory you see the number of logs that are deleted, but they are still taking up the space.

filebeat  2209 user 4r   REG  253,3 4147407173 77988311 /some/local/dummy/export/path/ratedata_2018-03-20T04-51-37.821.csv (deleted)

I have read through the documentation and am aware that filebeat will not close the harvester until it has finished reading the data. Once i restart the agent the file system goes down. I have used some of these settings on our config:

- input_type: log
  paths:

    - /my_path/*

  #Excludes .csv files
  exclude_files: ['\.csv']

  #Closes/Releases if file is not written to in 5 minutes.
  close_inactive: 5m

  scan_frequency: 10s

  tail_files: true

  enabled: true

  fields_under_root: true
  fields:
    service: my_service

Why is filebeat still holding onto the file?

Are there any other "best practice" settings i should be using?

Thanks for the help.

[ruflin]

Which Filebeat version are you using?

Could you share your filebeat log, best in debug mode? There you should see more details on why Filebeat decides to keep your file open.

[cchooks2 (Steven)]

We are using filebeat 5.5.

Attached is a snippet of the filebeat log on DBG:

The rest of the logs were the actual messages we are ingesting...

Thanks

[ruflin]

Thanks for the log. The above is the startup of the beat where files are going to be opened. The interesting part about closing files (or not) happens after the publishing of the events. If you send lots of events the problem is the closing of the file messages are hidden somewhere in between or after. Can you check your log file for messages which contain close?