Hi Elastic Community,
We have a bottleneck on CPU using filebeat. The service takes anywhere from 10-80% of CPU at all times.
After excluding certain paths, the CPU no longer spikes. In each of the path subdirectories exists 5000+ log files being captured
In the filebeat log files, I see some error messages which look like this:
2020-11-20T13:57:08.232-0600 ERROR log/input.go:501 Harvester could not be started on new file: C:\Logs\...\...\log.log.56, Err: error setting up harvester: Harvester setup failed. Unexpected file opening error: Failed opening C:\Logs\...\...\log.log.56: Error creating file 'C:\Logs\...\...\log.log.56': The system cannot find the file specified.
2020-11-17T16:53:10.466-0600 ERROR log/input.go:226 input state for C:\Logs\...\...\log.log was not removed: CreateFile C:\Logs\...\...\log.log: Access is denied.
2020-11-12T09:47:10.968-0600 ERROR log/harvester.go:368 Error updating file size: GetFileType C:\Logs\...\...\log.log: use of closed file; File: C:\Logs\...\...\log.log
After some initial research, I am considering
harvester_limit so filebeat doesn't throttle CPU so hard.
Has anyone experienced this issue before with filebeat across many log files?