Having trouble with filebeat not releasing rotated file

Elastic Stack Beats
1

We're having some trouble with filebeat not releasing files after logrotate is run.
Here is the filebeat conf:

filebeat:
  prospectors:
    -
      paths:
        - /home/vhosts/XXXX/current/log/production.log
      document_type: production
    -
      paths:
        - /var/log/alternatives.log
        - /var/log/apport.log
        - /var/log/auth.log
        - /var/log/bind.log
        - /var/log/boot.log
        - /var/log/btmp
        - /var/log/dhcpd.log
        - /var/log/diskmap.log
        - /var/log/faillog
        - /var/log/firewall.log
        - /var/log/kern.log
        - /var/log/lastlog
        - /var/log/mail.log
        - /var/log/syslog
      document_type: syslog
    -
      paths:
        - /var/log/nginx/error.log
      document_type: nginx_error
    -
      paths:
        - /var/log/nginx/access.log
      document_type: nginx
    -
      paths:
        - /var/log/haproxy.log
      document_type: haproxy
    -
      paths:
        - /var/log/ceph/*.log
      document_type: ceph

      input_type: log
      exclude_files: [".gz$",".1$"]
      scan_frequency: 5s
      ignore_older: 24h
      close_older: 5m
  registry_file: /var/lib/filebeat/registry
############################# Output ##########################################
output:
  logstash:
    hosts: ["10.130.192.99:9223","10.130.192.114:9223"]
    loadbalance: true
    worker: 1
    compression_level: 3
    loadbalance: true
    index: filebeat
############################# Shipper #########################################
shipper:
  tags: ["filebeat", "txc1","XXXX","US"]
  geoip:
    paths:
      - "/usr/share/GeoIP/GeoLiteCity.dat"
############################# Logging #########################################
logging:
  to_syslog: false
  to_files: true
  files:
    path: /var/log/mybeat
    name: filebeat
    rotateeverybytes: 10485760 # = 10MB
    keepfiles: 7
  level: debug

Particularly /var/log/firewall.log is kept open after it is renamed to /var/log/firewall.log.1

2

Which filebeat version are you using? As you haven't defined any ignore_older or close_older for this prospector, it should just stick with the default close_older. Is there anything special about the firewall.log file? Can you check the ModTime of firewall.log.1?

3

Thank you for your response.

root@txc1-fw1:/var/log# lsof ./* | grep filebeat | grep ".1"| awk '{print $9}'| while read l; do echo $l;lsof $l;stat -c "%y" $l;echo "===";done
./auth.log.1
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
filebeat 11931 root 10r REG 252,3 136860 170 ./auth.log.1
2016-05-16 00:17:01.754603091 +0100

./syslog.1
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
filebeat 11931 root 14r REG 252,3 525031 168 ./syslog.1
2016-05-16 00:17:01.754603091 +0100

./kern.log.1
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
filebeat 11931 root 17r REG 252,3 87781 169 ./kern.log.1
2016-05-15 19:25:36.313033918 +0100

I don't know if this info is of any help to you. As you can see its keeping a few *.log.1 files open, but drops them when we restart filebeat.

Edit-
Sorry forgot you asked for filebeat version:
filebeat version 1.1.2 (amd64)

4

For the above command, it would be important to know, what the time is you took the command.

Filebeat 1.1.2 does not support close_older, that was only introduced with Filebeat 1.2.