High CPU and Harvester Errors with 20000 log files

[jbudin (J B)]

Hi Elastic Community,

We have a bottleneck on CPU using filebeat. The service takes anywhere from 10-80% of CPU at all times.

After excluding certain paths, the CPU no longer spikes. In each of the path subdirectories exists 5000+ log files being captured \**\*.log*

In the filebeat log files, I see some error messages which look like this:
2020-11-20T13:57:08.232-0600 ERROR log/input.go:501 Harvester could not be started on new file: C:\Logs\...\...\log.log.56, Err: error setting up harvester: Harvester setup failed. Unexpected file opening error: Failed opening C:\Logs\...\...\log.log.56: Error creating file 'C:\Logs\...\...\log.log.56': The system cannot find the file specified.
2020-11-17T16:53:10.466-0600 ERROR log/input.go:226 input state for C:\Logs\...\...\log.log was not removed: CreateFile C:\Logs\...\...\log.log: Access is denied.
2020-11-12T09:47:10.968-0600 ERROR log/harvester.go:368 Error updating file size: GetFileType C:\Logs\...\...\log.log: use of closed file; File: C:\Logs\...\...\log.log

After some initial research, I am considering exclude_files: or harvester_limit so filebeat doesn't throttle CPU so hard.

Has anyone experienced this issue before with filebeat across many log files?

Thanks!
-JB

[shaunak (Shaunak Kashyap)]

I would suggest playing with the harvester_limit first and see what impact that has on the CPU, before you try the exclude_files option.

Shaunak

[jbudin (J B)]

Hi Shaunak,

Our .yaml file has a couple paths listed for each - type: log such as:
'C:\Logs\...\...\*.log*'
'C:\Logs\...\...\**\*.log*'

After removing one, and modifying the other to
'C:\Logs\...\...\...\*\*.log*'

We are no longer experiencing the CPU issue.

Thanks for the reply,
-JB