# On restart Harversters are opened for ignored files
**URL:** https://discuss.elastic.co/t/on-restart-harversters-are-opened-for-ignored-files/50038
**Category:** Beats
**Tags:** filebeat
**Created:** [May 13, 2016, 10:18pm UTC](https://discuss.elastic.co/t/on-restart-harversters-are-opened-for-ignored-files/50038 "2016-05-13T22:18:55Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@Stuart\_Fraser](https://discuss.elastic.co/u/Stuart_Fraser)
#### Post date: [May 13, 2016, 10:18pm UTC](https://discuss.elastic.co/t/on-restart-harversters-are-opened-for-ignored-files/50038/1 "2016-05-13T22:18:55Z")
Impact
Unable to restart filebeat as expected after ignoring a lot of files.
Description
When filebeat is restarted Harversters open every file even if they should be ignored by the ignore\_older setting.
Environment
filebeat-1.2.2
Unix Suse (SLES) 11.4 64 bit
Steps to Reproduce
1. Use the following filebeat.yml
2. Create a whole lot of old files using
bash
for i in {1..50000}; do touch --date="2009-11-23 19:43:57" oldfile$i.log; done
1. Make sure the registry file does not exist by using
rm filebeat.registry
1. Run filebeat using
filebeat -e -v -c filebeat.yml
1. Observe Harversters are opened for all the ignored files as well a a bunch of file descriptor errors such as:
2016/05/13 21:39:21.206773 log.go:113: INFO Harvester started for file: oldfile1.log
2016/05/13 21:39:21.206941 log.go:113: INFO Harvester started for file: oldfile10008.log
....
2016/05/13 21:39:21.795646 log.go:113: INFO Harvester started for file: oldfile17191.log
2016/05/13 21:39:21.795917 log.go:113: INFO Harvester started for file: oldfile17193.log
2016/05/13 21:39:21.796116 log.go:248: ERR Failed opening oldfile17195.log: open oldfile17195.log: too many open files
2016/05/13 21:39:21.796177 log.go:103: ERR Stop Harvesting. Unexpected file opening error: open oldfile17195.log: too many open files
2016/05/13 21:39:21.795437 log.go:113: INFO Harvester started for file: oldfile17186.log
2016/05/13 21:39:21.795556 log.go:113: INFO Harvester started for file: oldfile1719.log
2016/05/13 21:39:21.795905 log.go:113: INFO Harvester started for file: oldfile17194.log
2016/05/13 21:39:21.796044 log.go:248: ERR Failed opening oldfile17196.log: open oldfile17196.log: too many open files
2016/05/13 21:39:21.796364 log.go:103: ERR Stop Harvesting. Unexpected file opening error: open oldfile17196.log: too many open files
2016/05/13 21:39:21.796352 log.go:248: ERR Failed opening oldfile17198.log: open oldfile17198.log: too many open files
2016/05/13 21:39:21.796389 log.go:103: ERR Stop Harvesting. Unexpected file opening error: open oldfile17198.log: too many open files
2016/05/13 21:39:21.796477 log.go:248: ERR Failed opening oldfile172.log: open oldfile172.log: too many open files
2016/05/13 21:39:21.796504 log.go:103: ERR Stop Harvesting. Unexpected file opening error: open oldfile172.log: too many open files
2016/05/13 21:39:21.796479 log.go:248: ERR Failed opening oldfile17197.log: open oldfile17197.log: too many open files
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [May 17, 2016, 7:19am UTC](https://discuss.elastic.co/t/on-restart-harversters-are-opened-for-ignored-files/50038/2 "2016-05-17T07:19:26Z")
Thanks a lot for the detailed report and reproduction steps. I will look into this and report back here.
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [May 17, 2016, 8:32am UTC](https://discuss.elastic.co/t/on-restart-harversters-are-opened-for-ignored-files/50038/3 "2016-05-17T08:32:27Z")
I tried to reproduce this but so far without success. In my tests it skips the files because of ignore\_older (as it should). There is a log line directly after startup that states the ignore\_older time, can you share this line?
```auto
2016/05/17 08:29:55.743044 prospector.go:132: INFO Set ignore_older duration to 2h0m0s
```
---
### Author:  [@Stuart\_Fraser](https://discuss.elastic.co/u/Stuart_Fraser)
#### Post date: [May 17, 2016, 11:20am UTC](https://discuss.elastic.co/t/on-restart-harversters-are-opened-for-ignored-files/50038/4 "2016-05-17T11:20:18Z")
I may have missed a step ...
2a) Create at least 1 new file
touch outfileA.log
Any new file and all ignored older files are added to the registry. When you restart filebeat 1.2.2 opens all the older than files.
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [May 17, 2016, 11:52am UTC](https://discuss.elastic.co/t/on-restart-harversters-are-opened-for-ignored-files/50038/5 "2016-05-17T11:52:31Z")
Ok, now I can reproduce it. What I did is now:
- Start filebeat with all the ignore\_older files
- Stop filebeat
- Start filebeat again (without removing the state)
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [May 17, 2016, 12:32pm UTC](https://discuss.elastic.co/t/on-restart-harversters-are-opened-for-ignored-files/50038/6 "2016-05-17T12:32:53Z")
It is not directly a bug but was the expected behaviour to make sure, in case one of the ignore\_older files actually got new content, that the new content is read. But it is not nice so I opened a PR with a fix that checks if the offsets are the same: [https://github.com/elastic/beats/pull/1649](https://github.com/elastic/beats/pull/1649)
This is also related to the discussion here: [https://github.com/elastic/beats/issues/1600](https://github.com/elastic/beats/issues/1600)
---
### Author:  [@Stuart\_Fraser](https://discuss.elastic.co/u/Stuart_Fraser)
#### Post date: [May 17, 2016, 12:57pm UTC](https://discuss.elastic.co/t/on-restart-harversters-are-opened-for-ignored-files/50038/7 "2016-05-17T12:57:14Z")
Thanks!
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 5, 2017, 9:51pm UTC](https://discuss.elastic.co/t/on-restart-harversters-are-opened-for-ignored-files/50038/8 "2017-07-05T21:51:51Z")