Dealing with frequent logrotation - ignore_older seems to be ignored

Elastic Stack Beats
1

I am using filebeat to forward logs from a pair of Varnish servers, and these produce a fairly large amount of data pretty quickly.
Because we don't want to keep this stuff, I've set up a one-hour log-rotation schedule for the file, so I end up with a bunch of files like:

-rw-r--r-- 1 root root 25M Dec  3 09:56 /var/log/varnish/varnishncsa.log
-rw-r--r-- 1 root root 26M Dec  3 06:01 varnishncsa.log.20151203.1449122461
-rw-r--r-- 1 root root 29M Dec  3 07:01 varnishncsa.log.20151203.1449126061
-rw-r--r-- 1 root root 41M Dec  3 08:01 varnishncsa.log.20151203.1449129661
-rw-r--r-- 1 root root 66M Dec  3 09:01 varnishncsa.log.20151203.1449133261

My filebeat.yml is set at the top to ignore_older: 1h

filebeat:
  ignore_older: 1h

  prospectors:
    -
      type: log
      document_type: varnish
      paths:
        - "/var/log/varnish/*.log"

But I still frequently find that the disk has filled up because filebeat is holding files open. As soon as I restart the filebeat service, all that space comes back.

I'll try and update this post with more details the next time this happens, such as the open-file list for filebeat.

filebeat version 1.0.0 (amd64)
2

ignore_older should definitively close the files. Can you try force_close_files? This will close files on rotation. There is a small "issue" with it in 1.0.0, so it would be best to try one of the nightly builds: https://beats-nightlies.s3.amazonaws.com/index.html?prefix=filebeat/ Looking forward to get some more data on this.

3

I'll change that setting on both nodes, and stick a nightly on one of the two nodes and report back in a few hours when some data has accumulated.

Here's a the current content of the log Varnish log folder:

-rw-r--r-- 1 root root  54M Dec  3 12:25 varnishncsa.log
-rw-r--r-- 1 root root  66M Dec  3 09:01 varnishncsa.log.20151203.1449133261
-rw-r--r-- 1 root root  32M Dec  3 10:01 varnishncsa.log.20151203.1449136861
-rw-r--r-- 1 root root 126M Dec  3 11:01 varnishncsa.log.20151203.1449140461
-rw-r--r-- 1 root root 130M Dec  3 12:01 varnishncsa.log.20151203.1449144061

and the open files by filebeat:

lr-x------ 1 root root 64 Dec  3 10:14 5 -> /var/log/varnish/varnishncsa.log.20151203.1449136861
lr-x------ 1 root root 64 Dec  3 10:14 6 -> /var/log/varnish/varnishncsa.log.20151203.1449140461
lr-x------ 1 root root 64 Dec  3 11:01 7 -> /var/log/varnish/varnishncsa.log.20151203.1449144061
lr-x------ 1 root root 64 Dec  3 12:01 8 -> /var/log/varnish/varnishncsa.log

As a note to confirm: I've checked in Kibana, and all the data is there, so it's not like Filebeat is still processing the old files.

4

Looking at your config again, I think I spotted the problem. ignore_older should be under the prospector as it is prospector specific. You defined it globally. In your case as no ignore_older is found, the default of 24h is taken.

5

Oh my. That makes sense. So I assume it should be defined under the "-" that delimits the particular prospector:

filebeat:
  prospectors:
    -
      ignore_older: 2h
      type: log
      document_type: varnish
      paths:
        - "/var/log/varnish/*.log"

output:
6

That fixed it, thanks @ruflin. Apologies for the excitement :slight_smile: