Filebeat holding deleted files which consumes disk space

[arya]

Filebeat process in the server is holding the deleted files which are forwarded by logstash forwarder which makes the server disk space full. Currently a cron job is in place to restart filebeat at regular intervals. But team demands a permanent solution for this. Filebeat version we used is below.
filebeat version 1.2.3 (386)

Below is our filebeat.yml file.

filebeat:
      prospectors:
        -
          paths:
            - /var/log/apache/*.log

          input_type: log

          fields_under_root: true
          ignore_older: 1h
          close_older: 30m
          document_type: logs
        -

I have tried to enable close_removed flag as well. But that didn't help. Please help in resolving this issue.

[tudor (Tudor Golubenco)]

@arya For a more permanent solution, I recommend upgrading to Filebeat 5.4 (it's compatible with ES 2.x) and use the close_timeout option. Set it to a value higher than your rotation interval.

Also, consider that if Filebeat is holding onto files for so long, it means that the output is often blocked. Perhaps your Logstash/Elasticsearch clusters are under-provisioned?

[arya]

Thanks Tudor.

So version 1.2.3 doesn't support close_timeout option?
This problem is there only for files which are deleted. All the other files getting forwarded successfully. So that means Logstash/Elasticsearch clusters working as expected right?

[arya]

Hi ,

Will filebeat locks the files if logstash is down in the receiving end? I couldnot telnet from my filebeat server to logstash.

Regards,
Arya

[tudor (Tudor Golubenco)]

If close_timeout is not set, a blocking output can cause Filebeat to keep the files open forever, which means the OS can't delete them. Does that answer the question?

[arya]

yes.. thank you.