High CPU and Harvester Errors with 20000 log files

Hi Elastic Community,

We have a bottleneck on CPU using filebeat. The service takes anywhere from 10-80% of CPU at all times.

After excluding certain paths, the CPU no longer spikes. In each of the path subdirectories exists 5000+ log files being captured \**\*.log*

In the filebeat log files, I see some error messages which look like this:
2020-11-20T13:57:08.232-0600 ERROR log/input.go:501 Harvester could not be started on new file: C:\Logs\...\...\log.log.56, Err: error setting up harvester: Harvester setup failed. Unexpected file opening error: Failed opening C:\Logs\...\...\log.log.56: Error creating file 'C:\Logs\...\...\log.log.56': The system cannot find the file specified.
2020-11-17T16:53:10.466-0600 ERROR log/input.go:226 input state for C:\Logs\...\...\log.log was not removed: CreateFile C:\Logs\...\...\log.log: Access is denied.
2020-11-12T09:47:10.968-0600 ERROR log/harvester.go:368 Error updating file size: GetFileType C:\Logs\...\...\log.log: use of closed file; File: C:\Logs\...\...\log.log

After some initial research, I am considering exclude_files: or harvester_limit so filebeat doesn't throttle CPU so hard.

Has anyone experienced this issue before with filebeat across many log files?


I would suggest playing with the harvester_limit first and see what impact that has on the CPU, before you try the exclude_files option.


Hi Shaunak,

Our .yaml file has a couple paths listed for each - type: log such as:

After removing one, and modifying the other to

We are no longer experiencing the CPU issue.

Thanks for the reply,

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.