Filebeat High CPU usage - IIS Logs (Multiple Virtual Host)

Elastic Stack Beats
1

I have been trying to tune filebeat on an IIS server where it's peaking at 50-65% of the CPU every few seconds. This is on a Windows Server 2016 Datacenter, filebeat is managed by sidecar, feature IIS, with 47 virtual hosts, some having 2000 log files. I am on filebeat version 8.9. Do you have any suggestions with this setup of this many log files, how I can tune filebeat not to use so much processor? There is only one active log file in each virtual host folder being written to at a time. Thank you for any suggestions I may try.

Config:

output.logstash:
   hosts: ["10.3.3.3:5044"]
path:
   data: ${sidecar.spoolDir!"C:\\Program Files\\Graylog\\sidecar\\cache\\filebeat"}\data
   logs: ${sidecar.spoolDir!"C:\\Program Files\\Graylog\\sidecar"}\logs
tags:
- windows
filebeat.inputs:
- type: filestream
  id: iis
  enabled: true
  paths:
  - "C:/inetpub/logs/LogFiles/*/*.log"
  exclude_lines: ['^#']
#changed default scan frequency to 30s hoping to reduce harvesting, however I still see the filebeat process spiking at 50% every few seconds
  scan_frequency: 30s
#was hoping ignore older would prevent filebeat from reading and processing through all the older log files in the virtual host directory
  ignore_older: 1h
#configured close inactive to send harvesters for inactive log files 
  close_inactive: 30m
2

Howdy!

8.9.0 had its second birthday today, given that the release is 2 years old it might be advisable to upgrade to a more recent version to make sure you aren't troubleshooting an issue that has been resolved in the software.

That being said, the logs, especially if you enable debug logging, would be very useful to help figure out what activity is in progress when the usage spike happens.

Can you share a normal log and a debug log (run filebeat with filebeat -e -d "*"). The debug log will be extremely verbose so running it for just long enough to observe a spike or two should be sufficient.

3

You have a lot of logs there. Try to process older in few steps, by month/year or something like.

Another option is to reduce CPU number.

max_procs

Sets the maximum number of CPUs that can be executing simultaneously. The default is the number of logical CPUs available in the system.

If the CPU load is still high, after changing max_procs, read this topic. The blog about tuning Beats can be found here.

1 Like