Filebeat holding open handles on deleted files

Elastic Stack Beats
1

v. 6.8.4

We are experiencing issues daily with lingering open file handles on files which have been deleted by another process which has completed daily compression of logs.

The files seem to remain open forever until we manually restart the filebeat process.

Config example here:

- type: log
  enabled: true
  paths:
    - /logs/<app-name>/**/*.json.log
  fields:
    format: json
    ...more custom fields...

From the docs, close_inactive doesn't get included by default, but then it kinda does? I'm having a hard time confirming that.

When this option is enabled...
...The default is 5m.

If a file was "deleted" but then has no activity on it after being deleted, the handle should be closed if that "close_inactive" was defaulted to true. That deleted file should really have been deleted then, and the harvester should have no longer been able to find it....

I'm looking for confirmation on the behavior of the process when close_inactive is not set at all for any of the inputs.

2

close_inactive is enabled by default. However, it does not work as expected in 6.8.

Filebeat usually keeps files open if there is backpressure in the output, e.g. Elasticsearch is not able to process incoming events. Are there any issues in your output causing the Beat keeping files open?

The behaviour of this option has been fixed in Filebeat 7.x. Also, we have come up with a new input for log files called filestream: filestream input | Filebeat Reference [7.14] | Elastic It has much better support for closing files in time even if there is backpressure in the system. Please consider switching versions and inputs.

3

Hi Noémi - Thanks for the confirmation.

The output is set to logstash. We're pretty sure there is back pressure there, but are having a tough time confirming due to lack of visibility. The metrics available for filebeat are showing no errors/retries occurring, and logstash metrics/logs aren't showing anything about rejecting requests either.

Can you provide more details on "it does not work as expected in 6.8" ?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.