Tested on 6.8.4 and 6.8.15.
This issue seems to crop up in applications with a high rate of logs.
Takes a day or two to get to that point, noticed the file handles count increasing steadily throughout.
[root@ldap-master001-prod]$ ps aux | grep [f]ilebeat
root 58474 0.1 0.5 343600 44272 ? Ssl Apr20 2:35 /opt/filebeat-6.8.15-linux-x86_64/filebeat -c /opt/filebeat-6.8.15-linux-x86_64/filebeat-primary.yml
[root@ldap-master001-prod]$ sudo ls -l /proc/58474/fd
total 0
lr-x------ 1 root root 64 Apr 21 17:30 0 -> /dev/null
lrwx------ 1 root root 64 Apr 21 17:30 1 -> socket:[1981213191]
lr-x------ 1 root root 64 Apr 21 17:30 10 -> /var/log/slapd/ldap.log.1 (deleted)
lr-x------ 1 root root 64 Apr 21 17:30 11 -> /var/log/slapd/ldap.log.1 (deleted)
lr-x------ 1 root root 64 Apr 21 17:30 12 -> /var/log/slapd/ldap.log.1 (deleted)
lr-x------ 1 root root 64 Apr 21 17:30 13 -> /var/log/slapd/ldap.log.1 (deleted)
lr-x------ 1 root root 64 Apr 21 17:30 14 -> /var/log/slapd/ldap.log.1 (deleted)
lr-x------ 1 root root 64 Apr 21 17:30 15 -> /var/log/slapd/ldap.log.1 (deleted)
lr-x------ 1 root root 64 Apr 21 17:30 16 -> /var/log/slapd/ldap.log.1 (deleted)
lr-x------ 1 root root 64 Apr 21 17:30 17 -> /var/log/slapd/ldap.log.1 (deleted)
lr-x------ 1 root root 64 Apr 21 17:30 18 -> /var/log/slapd/ldap.log.1 (deleted)
lr-x------ 1 root root 64 Apr 21 17:30 19 -> /var/log/slapd/ldap.log.1 (deleted)
lrwx------ 1 root root 64 Apr 21 17:30 2 -> socket:[1981213191]
lr-x------ 1 root root 64 Apr 21 17:30 20 -> /var/log/slapd/ldap.log.1 (deleted)
lr-x------ 1 root root 64 Apr 21 17:30 21 -> /var/log/slapd/ldap.log.1 (deleted)
lr-x------ 1 root root 64 Apr 21 17:30 22 -> /var/log/slapd/ldap.log.1 (deleted)
lr-x------ 1 root root 64 Apr 21 17:30 23 -> /var/log/slapd/ldap.log.1 (deleted)
lr-x------ 1 root root 64 Apr 21 17:30 24 -> /var/log/slapd/ldap.log.1 (deleted)
lr-x------ 1 root root 64 Apr 21 17:30 25 -> /var/log/slapd/ldap.log.1 (deleted)
lr-x------ 1 root root 64 Apr 21 17:30 26 -> /var/log/slapd/ldap.log.1 (deleted)
lr-x------ 1 root root 64 Apr 21 17:30 27 -> /var/log/slapd/ldap.log.1 (deleted)
lr-x------ 1 root root 64 Apr 21 17:30 28 -> /var/log/slapd/ldap.log.1 (deleted)
lr-x------ 1 root root 64 Apr 21 17:30 29 -> /var/log/slapd/ldap.log.1 (deleted)
l-wx------ 1 root root 64 Apr 21 17:30 3 -> /var/log/filebeat-iu_dev/filebeat
lr-x------ 1 root root 64 Apr 21 17:30 30 -> /var/log/slapd/ldap.log.1 (deleted)
lr-x------ 1 root root 64 Apr 21 17:30 31 -> /var/log/slapd/ldap.log.1 (deleted)
lr-x------ 1 root root 64 Apr 21 17:30 32 -> /var/log/slapd/ldap.log.1 (deleted)
lr-x------ 1 root root 64 Apr 21 17:30 33 -> /var/log/slapd/ldap.log.1 (deleted)
lr-x------ 1 root root 64 Apr 21 17:30 34 -> /var/log/slapd/ldap.log.1
lr-x------ 1 root root 64 Apr 21 17:30 35 -> /var/log/slapd/ldap.log
lrwx------ 1 root root 64 Apr 21 17:30 4 -> anon_inode:[eventpoll]
lr-x------ 1 root root 64 Apr 21 17:30 5 -> /var/log/slapd/ldap.log.1 (deleted)
lrwx------ 1 root root 64 Apr 21 17:30 6 -> socket:[1981213198]
lr-x------ 1 root root 64 Apr 21 17:30 7 -> /var/log/slapd/ldap.log.1 (deleted)
lr-x------ 1 root root 64 Apr 21 17:30 8 -> /var/log/slapd/ldap.log.1 (deleted)
lr-x------ 1 root root 64 Apr 21 17:30 9 -> /var/log/slapd/ldap.log.1 (deleted)
Our inputs config:
...
filebeat.inputs:
- type: log
paths: [ "/var/log/slapd/ldap.log" ]
fields:
component: "ldap"
data_type: "slapd"
cluster: "primary"
...
This is how the log directory looks:
[root@ldap-master001-prod]$ sudo ls /var/log/slapd/
ldap.log
ldap.log.1
ldap.log.2.gz
ldap.log.3.gz
...
ldap.log.97.gz