# Harvesting specified number of most recent files **URL:** \ **Category:** Beats\ **Created:** [July 4, 2018, 11:17am UTC](https://discuss.elastic.co/t/harvesting-specified-number-of-most-recent-files/138544 "2018-07-04T11:17:45Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![kskubala](https://avatars.discourse-cdn.com/v4/letter/k/c5a1d2/32.png) [@kskubala](https://discuss.elastic.co/u/kskubala)\ **Post date:** [July 4, 2018, 11:17am UTC](https://discuss.elastic.co/t/harvesting-specified-number-of-most-recent-files/138544/1 "2018-07-04T11:17:45Z")
Hi, I am wondering how to harvest only specified number of most recent files ? Or for example is there any way to not harvest files that have been renamed (but they always need to be read to the end so I think _close\_renamed_ wont work). I have so huge amount of log files that I need to limit harvested files. Every hour there is created new file in which logs are being stored (after - it is renamed). I need this new data in ES but can't miss logs that are a bit older. I tried ignore\_older but I don't have warranty that nothing is missed. File older than ignore\_older won't be ever read, am I right ? ---
**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\ **Post date:** [July 4, 2018, 2:51pm UTC](https://discuss.elastic.co/t/harvesting-specified-number-of-most-recent-files/138544/2 "2018-07-04T14:51:03Z")
Hi @kskubala, I don't completely understand your problem. Even if you create a file every hour filebeat should be able to harvest all of them. When [ignore\_older](https://www.elastic.co/guide/en/beats/filebeat/6.3/filebeat-input-log.html#filebeat-input-log-ignore-older) option is enabled filebeat ignores files modified before the specified timespan. ---
**Author:** ![kskubala](https://avatars.discourse-cdn.com/v4/letter/k/c5a1d2/32.png) [@kskubala](https://discuss.elastic.co/u/kskubala)\ **Post date:** [July 5, 2018, 6:51am UTC](https://discuss.elastic.co/t/harvesting-specified-number-of-most-recent-files/138544/3 "2018-07-05T06:51:51Z")
I will explain it step by step. I have many files that needs to be harvested. I want them to be put in elastic in real time. So I want all the most recent logs that are written to files to be almost immediately stored in ES. When I have many files, scanning files and harvesting many of them is causing that I don't have recent logs in ES. As a solution I have tried to set ignore\_older, but I am worried that some logs will be missed (some logs are written to file once per hour). Setting higher value of ignore\_older is causing same thing - no real time data. ---
**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\ **Post date:** [July 5, 2018, 7:42am UTC](https://discuss.elastic.co/t/harvesting-specified-number-of-most-recent-files/138544/4 "2018-07-05T07:42:40Z")
@kskubala could you share the configuration you are using for these files? ---
**Author:** ![kskubala](https://avatars.discourse-cdn.com/v4/letter/k/c5a1d2/32.png) [@kskubala](https://discuss.elastic.co/u/kskubala)\ **Post date:** [July 5, 2018, 10:08am UTC](https://discuss.elastic.co/t/harvesting-specified-number-of-most-recent-files/138544/5 "2018-07-05T10:08:46Z")
Unfortunately I can't. Anyway there isn't anything according to harvesting settings except for > ignore\_older: 5m ---
**Author:** ![jsoriano](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jsoriano/32/27920_2.png) [@jsoriano](https://discuss.elastic.co/u/jsoriano)\ **Post date:** [July 5, 2018, 3:30pm UTC](https://discuss.elastic.co/t/harvesting-specified-number-of-most-recent-files/138544/6 "2018-07-05T15:30:16Z")
Take a look also to the `close_inactive` option. If `ignore_older` is set, it has to be set to a greater value than `close_inactive` that defaults to 5m. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 2, 2018, 5:30pm UTC](https://discuss.elastic.co/t/harvesting-specified-number-of-most-recent-files/138544/7 "2018-08-02T17:30:17Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.