Holding onto handles of deleted files

[justinyeh3112 (Justin Yeh)]

Hi,
Is there a good way to have filebeat let go of the filehandle for a file after a certain amount of inactivity to the file?

We are running into the following problem:

1. Filebeat is tracking FileA
2. The output target goes down, so Filebeat is unable to ship any additional log lines for FileA
3. FileA is deleted (rotated, disk cleanup, etc)
4. Since Filebeat still holds onto the filehandle, the OS never reclaims that disk space.

We have a few systems that produce a large number of temporary logs with very verbose log lines and when the above happens, over time, disk space can get filled up. In this scenario, if we have to choose between disk space filling and the service failing, or, failing to ship some log lines, we'd rather choose the latter.

It seems to me that close_inactive should kick in after the default of 5m and allow the file to be closed, but that doesn't seem to be working. We've also tried setting close_timeout with no luck.

[ruflin]

Having the output blocked, means also all of the close_* options (https://www.elastic.co/guide/en/beats/filebeat/5.1/configuration-filebeat-options.html#close-options) will not apply as the harvesters are completely blocked. But we are definitively thinking about solutions to this: https://github.com/elastic/beats/pull/3091 But we stoped some initial approaches as we want to have a solid solution here.