We want to use metricbeat to monitor processes on our Windows servers.
The problem is that we have 200+ processes which seems to cause peformance issues:
WmiPrvSE.execonsumes 90%+ of one CPU core.
- Metricbeat freezes completely for first couple of minutes after starting the processes.
- At least when using Topbeat 1.3.1 we see that its log is full of
WARN Ignoring tick(s) due to processing taking longer than one period(we have
period: 10s). Not sure if Metricbeat 5.2.1 still has the problem.
We see that the high CPU usage is caused by obtaining processes' command line via WMI (as discussed in Windows Topbeat Causes High CPU Utilization in WMI Provider Host [Solved]).
It would be great if Metricbeat somehow allowed user to specify (s)he does not need the
system.process.cmdline in order to avoid that dramatic WMI performance overhead.
Note this does not have the desired effect:
fields: [ system.process.cmdline ]
Also note it may be the case that the problem only manifests itself when processes have pretty long command lines (which our PROD processes do). So, the
spawn_dummy_processes.bat batch file given below has some quick-and-dirty procedure of spawning processes with long command lines.
As a proof-of-concept we've patched the Metricbeat to not query WMI. Here are the results:
spawn_dummy_processes.bat to use for testing
call :generateRandomString 300
echo Spawning %childrenCount% processes that will live for %childrenLifeDuration% seconds...
FOR /L %%X IN (1,1,%childrenCount%) DO call :spawn %%X
echo Press any key to terminate spawned processes prematurely...
pause > NUL
echo Killing processes...
FOR /L %%X IN (1,1,%childrenCount%) DO waitfor /SI XXX%%X > NUL 2>&1
::call :generateRandomString 400
:: add randomized preffix, so that processes have different command lines
set longString=%random% %longString%
start /MIN cmd "/c echo %longString%& waitfor /T %childrenLifeDuration% XXX%1"
::waitfor /T 1 ZZZZ > NUL 2>&1
call :generateRandomStringImpl "longString" %1
IF 0==%remainingSteps% GOTO :EOF
set /A remainingSteps = %remainingSteps% - 1
set accumString=%accumString% %random% %random% %random%
call :generateRandomStringImpl "%accumString%" %remainingSteps%