# High CPU and Harvester Errors with 20000 log files

**URL:** <https://discuss.elastic.co/t/high-cpu-and-harvester-errors-with-20000-log-files/263290>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [February 4, 2021, 7:05pm UTC](https://discuss.elastic.co/t/high-cpu-and-harvester-errors-with-20000-log-files/263290 "2021-02-04T19:05:10Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![jbudin](https://avatars.discourse-cdn.com/v4/letter/j/f07891/32.png) [@jbudin](https://discuss.elastic.co/u/jbudin)\
**Post date:** [February 4, 2021, 7:05pm UTC](https://discuss.elastic.co/t/high-cpu-and-harvester-errors-with-20000-log-files/263290/1 "2021-02-04T19:05:10Z")

</div>

Hi Elastic Community,

We have a bottleneck on CPU using filebeat. The service takes anywhere from 10-80% of CPU at all times.

After excluding certain paths, the CPU no longer spikes. In each of the path subdirectories exists 5000+ log files being captured `\**\*.log*`

In the filebeat log files, I see some error messages which look like this:  
`2020-11-20T13:57:08.232-0600	ERROR	log/input.go:501	Harvester could not be started on new file: C:\Logs\...\...\log.log.56, Err: error setting up harvester: Harvester setup failed. Unexpected file opening error: Failed opening C:\Logs\...\...\log.log.56: Error creating file 'C:\Logs\...\...\log.log.56': The system cannot find the file specified.`  
`2020-11-17T16:53:10.466-0600	ERROR	log/input.go:226	input state for C:\Logs\...\...\log.log was not removed: CreateFile C:\Logs\...\...\log.log: Access is denied.`  
`2020-11-12T09:47:10.968-0600	ERROR	log/harvester.go:368	Error updating file size: GetFileType C:\Logs\...\...\log.log: use of closed file; File: C:\Logs\...\...\log.log`

After some initial research, I am considering `exclude_files:` or `harvester_limit` so filebeat doesn't throttle CPU so hard.

Has anyone experienced this issue before with filebeat across many log files?

Thanks!  
-JB

---

<div class="post-metadata">

**Author:** ![shaunak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaunak/32/6643_2.png) [@shaunak](https://discuss.elastic.co/u/shaunak)\
**Post date:** [February 4, 2021, 10:00pm UTC](https://discuss.elastic.co/t/high-cpu-and-harvester-errors-with-20000-log-files/263290/2 "2021-02-04T22:00:06Z")

</div>

I would suggest playing with the `harvester_limit` first and see what impact that has on the CPU, before you try the `exclude_files` option.

Shaunak

---

<div class="post-metadata">

**Author:** ![jbudin](https://avatars.discourse-cdn.com/v4/letter/j/f07891/32.png) [@jbudin](https://discuss.elastic.co/u/jbudin)\
**Post date:** [February 4, 2021, 10:06pm UTC](https://discuss.elastic.co/t/high-cpu-and-harvester-errors-with-20000-log-files/263290/3 "2021-02-04T22:06:16Z")

</div>

Hi Shaunak,

Our .yaml file has a couple paths listed for each `- type: log` such as:  
`'C:\Logs\...\...\*.log*'`  
`'C:\Logs\...\...\**\*.log*'`

After removing one, and modifying the other to  
`'C:\Logs\...\...\...\*\*.log*'`

We are no longer experiencing the CPU issue.

Thanks for the reply,  
-JB

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 5, 2021, 12:06am UTC](https://discuss.elastic.co/t/high-cpu-and-harvester-errors-with-20000-log-files/263290/4 "2021-03-05T00:06:58Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.