High CPU usage for auditbeat

Elastic Stack Beats
1

Hi, I am trying to use auditbeat to collect Linux Audit log and send them to our elasticsearch server.
In order to compare performance with auditd, I tried the following configuration:

auditbeat.modules:

-module: auditd
audit_rules: |
-a always,exit -F arch=b64 -S all -k exec

-module: file_integrity
paths:
-/bin
-/usr/bin
-/sbin
-/usr/sbin
-/etc

output.file:
path: "/tmp/auditbeat"
filename: auditbeat

With the same configuration for auditd, I found that auditbeat has much higher CPU usage. In particular, auditbeat takes above 14% CPU usage while auditd take maximum 5%. The auditd version is auditbeat-6.2.4-amd64.

Are there any ways to reduce the CPU usages as auditd?

2

I think your audit rule is too broad, -S all without further filtering will capture ALL syscalls made by all processes. This can easily generate thousands of events per second and is hardly what you want. The common usage for -S all is pairing it with -F arch=b32 so that the usage of a 32-bit API is flagged.

Try narrowing down your rules to something more manageable. The example rules in the configuration are a good start.

As the difference in performance between auditbeat and auditd, note that auditd just writes the raw events to a log file. Auditbeat does plenty of post-processing to present the events in a meaningful way that can be further analyzed in Elasticsearch.

3

Have the same problem.
Auditbeat system module consumes about 12% average of the Intel(R) Xeon(R) CPU E5-2695 v3 @ 2.30GHz.
This is a huge amount for a prod server. It's to expensive to use it in prod.
Have to say auditbeat "audit" module consumes about 3% even with a huge set of syscalls monitoring.
But why does auditbeat "system" module consume such an amount of CPU?
It looks like non optimized algorithm in software...

4

@dsv, there's been some CPU issues with the system module at least in 7.7 and 7.8.0. Which version are you running?

Do you observe one of the logical CPUs going to 100% usage or is the load distributed between all CPUs?

5

auditbeat-7.8.0-1 (x86_64)

6

I don't know how to determine if 1 core used or not.
Every 10s CPU utilization jumps from 0 to ~102% and corresponding TIME in top utility increases to about 1s by every spike.
If i add the "-socket" option utilization sometimes spikes to 200%.
I changed the "-period" from 10s to 60s and this is temporary workaround for me (load is about 4%).
But i see the auditbeat configuration with 60s "- period" sometimes after restart consumes about 20% (average).
And the unstable behavior of the CPU load doesn't allow me still to deploy auditbeat in production.

7

You can see for example using htop -p $(pidof auditbeat).

Anyway, I suggest you try the custom 7.8.0 snapshot build in this post:

If it solves the problem, this fix will be released in the upcoming 7.8.1