# Harvester\_limit over directory with more 1M files

**URL:** <https://discuss.elastic.co/t/harvester-limit-over-directory-with-more-1m-files/128412>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [April 17, 2018, 6:01pm UTC](https://discuss.elastic.co/t/harvester-limit-over-directory-with-more-1m-files/128412 "2018-04-17T18:01:06Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Juan\_Andres\_Ramirez](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/juan_andres_ramirez/32/9467_2.png) [@Juan\_Andres\_Ramirez](https://discuss.elastic.co/u/Juan_Andres_Ramirez)\
**Post date:** [April 17, 2018, 6:01pm UTC](https://discuss.elastic.co/t/harvester-limit-over-directory-with-more-1m-files/128412/1 "2018-04-17T18:01:06Z")

</div>

hello guys,  
I have a directory with around 1M files, so I tried added in the filebeat.yml a harvester\_limit:

```
filebeat.prospectors:
- type: log
  enabled: true
  paths:
    - /logs/**
  harvester_limit: 400000

```

But filebeat trying harvester all files in the directory, so I have errors when harvester over 400000:

`ERROR	log/prospector.go:437	Harvester could not be started on new file:file-1.log, Err: Harvester limit reached`

So, I have some questions:

1- This is a really error?  
2- What happen with these files with error harvester limit?, filebeat will try harvester these files again?

Thank you.

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [April 18, 2018, 12:53pm UTC](https://discuss.elastic.co/t/harvester-limit-over-directory-with-more-1m-files/128412/2 "2018-04-18T12:53:13Z")

</div>

From the harvesters point of view it's an error, because it cannot be started due to the configured limit. From users' point of view I would say it's rather a warning. You can choose to act on it and increase the harvester\_limit if required. But if it's on purpose it might be a bit annoying to see Filebeat send these error messages.

Files which cannot have a harvester due to the limit is not read. By specifying a limit you tell Filebeat to read at most 400,000 files in parallel. If one of those files is read completely and closed a new harvester can be started for a new file. Filebeat scans the directory for unread files periodically. (So the answer is to your second question is yes.) The frequency can be set using `scan_frequency`. See more on this option: [https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html#scan-frequency](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html#scan-frequency)  
In theory I can imagine a situation when you have 400,000 log files which is updated all the time and Filebeat cannot keep up with and those files are never closed, so the other 600,000 log files can never be read. But in real life I think the log flow is not that fast.  
Also, to avoid keeping log files open for too long, you can set `close_inactive`. See more on this option here: [https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html#close-inactive](https://www.elastic.co/guide/en/beats/filebeat/current/configuration-filebeat-options.html#close-inactive)

---

<div class="post-metadata">

**Author:** ![Juan\_Andres\_Ramirez](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/juan_andres_ramirez/32/9467_2.png) [@Juan\_Andres\_Ramirez](https://discuss.elastic.co/u/Juan_Andres_Ramirez)\
**Post date:** [April 18, 2018, 1:10pm UTC](https://discuss.elastic.co/t/harvester-limit-over-directory-with-more-1m-files/128412/3 "2018-04-18T13:10:25Z")

</div>

Understood, thanks.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 16, 2018, 1:10pm UTC](https://discuss.elastic.co/t/harvester-limit-over-directory-with-more-1m-files/128412/4 "2018-05-16T13:10:25Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.