Hi,
I encountered the issue when using filebeat with logstash. Due to firewall configuration mistake, filebeat could not connect and send logs. After some time, the host run out of space in /var/log - filebeat was keeping old log files open even though logrotate has already removed them long time ago.
This was with pretty old filebeat client (1.2.3) so I wanted to upgrade to 5.2.1 and use close_removed to fix the issue. After I tested this a bit I found out that the behavior of 5.2.1 also seems to be incorrect. Here's my test scenario:
- Started with new fully updated ubuntu 14.04 server and installed filebeat 5.2.1 deb from official download page
- Changed default configuration: read only /var/log/syslog, increase log verbosity to debug, change output type to logstash
- Restarted filebeat and ensured it can't connect to logstash (by default it tries localhost)
- Forced few logrotates: logrotate -f /etc/logrotate.conf; for i in {1..1000}; do logger $i; done
After few forced rotations, filebeat keeps only removed syslog files open and seems to ignore new ones:
root@test:~# lsof -p 3408
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
filebeat 3408 root cwd DIR 253,1 4096 2 /
filebeat 3408 root rtd DIR 253,1 4096 2 /
filebeat 3408 root txt REG 253,1 15065599 148342 /usr/share/filebeat/bin/filebeat
filebeat 3408 root 0r CHR 1,9 0t0 5309 /dev/urandom
filebeat 3408 root 1u REG 253,1 687823 148333 /var/log/filebeat/filebeat
filebeat 3408 root 3r REG 253,1 32141 3007 /var/log/syslog.1 (deleted)
filebeat 3408 root 4u 0000 0,9 0 5259 anon_inode
filebeat 3408 root 5r REG 253,1 32000 28535 /var/log/syslog.1 (deleted)
Is there any way to forcibly close those old rotated files and focus on newer ones instead?
Regards,
Bartek