Filebeat and disk space

Elastic Stack Beats
1

Hi,
I'm using filebeat 1.2.3-1 and it generally works fine on most of our servers.
Some of our servers log files using log4j and its log rotation mechanism. It looks like filebeat never closes the file handle for the last file in the logrotation. Meaning - if the rotation is for 7 files for example, the seventh file which is being deleted is always kept open by filebeat:

lr-x------ 1 root root 64 Sep 26 07:06 14 -> /home/XXX/logs/XX.log.7 (deleted)
lr-x------ 1 root root 64 Sep 26 07:06 14 -> /home/XXX/logs/XX.log.7 (deleted)
lr-x------ 1 root root 64 Sep 26 07:06 14 -> /home/XXX/logs/XX.log.7 (deleted)
lr-x------ 1 root root 64 Sep 26 07:06 14 -> /home/XXX/logs/XX.log.7 (deleted)

This list keeps growing all the time

This happens even when the following parameters are set as follows:
ignore_older: 20m
close_older: 30m

These are linux servers, I didn't want to set the parameter: force_close_files to true.

As a result, our disk usage keeps growing continuously until the disk is full.
I didn't find any solution online for this issue besides the parameters mentioned above which didn't help.

Thanks for your advice!
Ofer

P.S - can an upgrade to filebeat 5.0 beta1 solve this issue? I didn't see anything related in the release notes.

1 Like
2

There are some cases in which in 1.x version in can come to race conditions which can also lead to open file handlers. All these should be solved in the 5.0 release as the state handling and open file handling was rewritten to solve all this issues. Upgrading to the 5.0 release should definitively solve these issues.

3

Thanks!
I will upgrade to 5.0 and see if that solves the issue.

4

This topic was automatically closed after 21 days. New replies are no longer allowed.