Logrotate + close_removed issue


I encountered the issue when using filebeat with logstash. Due to firewall configuration mistake, filebeat could not connect and send logs. After some time, the host run out of space in /var/log - filebeat was keeping old log files open even though logrotate has already removed them long time ago.

This was with pretty old filebeat client (1.2.3) so I wanted to upgrade to 5.2.1 and use close_removed to fix the issue. After I tested this a bit I found out that the behavior of 5.2.1 also seems to be incorrect. Here's my test scenario:

  • Started with new fully updated ubuntu 14.04 server and installed filebeat 5.2.1 deb from official download page
  • Changed default configuration: read only /var/log/syslog, increase log verbosity to debug, change output type to logstash
  • Restarted filebeat and ensured it can't connect to logstash (by default it tries localhost)
  • Forced few logrotates: logrotate -f /etc/logrotate.conf; for i in {1..1000}; do logger $i; done

After few forced rotations, filebeat keeps only removed syslog files open and seems to ignore new ones:

root@test:~# lsof -p 3408
filebeat 3408 root cwd DIR 253,1 4096 2 /
filebeat 3408 root rtd DIR 253,1 4096 2 /
filebeat 3408 root txt REG 253,1 15065599 148342 /usr/share/filebeat/bin/filebeat
filebeat 3408 root 0r CHR 1,9 0t0 5309 /dev/urandom
filebeat 3408 root 1u REG 253,1 687823 148333 /var/log/filebeat/filebeat
filebeat 3408 root 3r REG 253,1 32141 3007 /var/log/syslog.1 (deleted)
filebeat 3408 root 4u 0000 0,9 0 5259 anon_inode
filebeat 3408 root 5r REG 253,1 32000 28535 /var/log/syslog.1 (deleted)

Is there any way to forcibly close those old rotated files and focus on newer ones instead?


The above is the expected behaviour as filebeat wants to guarantee, that each file is sent at least once. Recently this PR (https://github.com/elastic/beats/pull/3511) was merged into master in case you want to release file handler when your output is stuck.

It's not yet in a release but you try out the nightly builds to test the feature: https://beats-nightlies.s3.amazonaws.com/index.html?prefix=filebeat/ Let me know if it works as expected.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.