# Dealing with frequent logrotation - ignore\_older seems to be ignored
**URL:** https://discuss.elastic.co/t/dealing-with-frequent-logrotation-ignore-older-seems-to-be-ignored/36271
**Category:** Beats
**Tags:** filebeat
**Created:** [December 3, 2015, 10:18am UTC](https://discuss.elastic.co/t/dealing-with-frequent-logrotation-ignore-older-seems-to-be-ignored/36271 "2015-12-03T10:18:05Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@Cylindric](https://discuss.elastic.co/u/Cylindric)
#### Post date: [December 3, 2015, 10:18am UTC](https://discuss.elastic.co/t/dealing-with-frequent-logrotation-ignore-older-seems-to-be-ignored/36271/1 "2015-12-03T10:18:05Z")
I am using filebeat to forward logs from a pair of Varnish servers, and these produce a fairly large amount of data pretty quickly.
Because we don't want to keep this stuff, I've set up a one-hour log-rotation schedule for the file, so I end up with a bunch of files like:
```
-rw-r--r-- 1 root root 25M Dec 3 09:56 /var/log/varnish/varnishncsa.log
-rw-r--r-- 1 root root 26M Dec 3 06:01 varnishncsa.log.20151203.1449122461
-rw-r--r-- 1 root root 29M Dec 3 07:01 varnishncsa.log.20151203.1449126061
-rw-r--r-- 1 root root 41M Dec 3 08:01 varnishncsa.log.20151203.1449129661
-rw-r--r-- 1 root root 66M Dec 3 09:01 varnishncsa.log.20151203.1449133261
```
My filebeat.yml is set at the top to ignore\_older: 1h
```
filebeat:
ignore_older: 1h
prospectors:
-
type: log
document_type: varnish
paths:
- "/var/log/varnish/*.log"
```
But I still frequently find that the disk has filled up because filebeat is holding files open. As soon as I restart the filebeat service, all that space comes back.
I'll try and update this post with more details the next time this happens, such as the open-file list for filebeat.
```
filebeat version 1.0.0 (amd64)
```
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [December 3, 2015, 12:18pm UTC](https://discuss.elastic.co/t/dealing-with-frequent-logrotation-ignore-older-seems-to-be-ignored/36271/2 "2015-12-03T12:18:43Z")
ignore\_older should definitively close the files. Can you try force\_close\_files? This will close files on rotation. There is a small "issue" with it in 1.0.0, so it would be best to try one of the nightly builds: [https://beats-nightlies.s3.amazonaws.com/index.html?prefix=filebeat/](https://beats-nightlies.s3.amazonaws.com/index.html?prefix=filebeat/) Looking forward to get some more data on this.
---
### Author:  [@Cylindric](https://discuss.elastic.co/u/Cylindric)
#### Post date: [December 3, 2015, 12:27pm UTC](https://discuss.elastic.co/t/dealing-with-frequent-logrotation-ignore-older-seems-to-be-ignored/36271/3 "2015-12-03T12:27:25Z")
I'll change that setting on both nodes, and stick a nightly on one of the two nodes and report back in a few hours when some data has accumulated.
Here's a the current content of the log Varnish log folder:
```
-rw-r--r-- 1 root root 54M Dec 3 12:25 varnishncsa.log
-rw-r--r-- 1 root root 66M Dec 3 09:01 varnishncsa.log.20151203.1449133261
-rw-r--r-- 1 root root 32M Dec 3 10:01 varnishncsa.log.20151203.1449136861
-rw-r--r-- 1 root root 126M Dec 3 11:01 varnishncsa.log.20151203.1449140461
-rw-r--r-- 1 root root 130M Dec 3 12:01 varnishncsa.log.20151203.1449144061
```
and the open files by filebeat:
```
lr-x------ 1 root root 64 Dec 3 10:14 5 -> /var/log/varnish/varnishncsa.log.20151203.1449136861
lr-x------ 1 root root 64 Dec 3 10:14 6 -> /var/log/varnish/varnishncsa.log.20151203.1449140461
lr-x------ 1 root root 64 Dec 3 11:01 7 -> /var/log/varnish/varnishncsa.log.20151203.1449144061
lr-x------ 1 root root 64 Dec 3 12:01 8 -> /var/log/varnish/varnishncsa.log
```
As a note to confirm: I've checked in Kibana, and all the data is there, so it's not like Filebeat is still processing the old files.
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [December 3, 2015, 2:50pm UTC](https://discuss.elastic.co/t/dealing-with-frequent-logrotation-ignore-older-seems-to-be-ignored/36271/4 "2015-12-03T14:50:26Z")
Looking at your config again, I think I spotted the problem. ignore\_older should be under the prospector as it is prospector specific. You defined it globally. In your case as no ignore\_older is found, the default of 24h is taken.
---
### Author:  [@Cylindric](https://discuss.elastic.co/u/Cylindric)
#### Post date: [December 3, 2015, 2:54pm UTC](https://discuss.elastic.co/t/dealing-with-frequent-logrotation-ignore-older-seems-to-be-ignored/36271/5 "2015-12-03T14:54:48Z")
Oh my. That makes sense. So I assume it should be defined under the "-" that delimits the particular prospector:
```
filebeat:
prospectors:
-
ignore_older: 2h
type: log
document_type: varnish
paths:
- "/var/log/varnish/*.log"
output:
```
---
### Author:  [@Cylindric](https://discuss.elastic.co/u/Cylindric)
#### Post date: [December 3, 2015, 3:33pm UTC](https://discuss.elastic.co/t/dealing-with-frequent-logrotation-ignore-older-seems-to-be-ignored/36271/6 "2015-12-03T15:33:33Z")
That fixed it, thanks @ruflin. Apologies for the excitement 🙂
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 5, 2017, 9:57pm UTC](https://discuss.elastic.co/t/dealing-with-frequent-logrotation-ignore-older-seems-to-be-ignored/36271/7 "2017-07-05T21:57:43Z")