Limit CPU/Memory usage in Auditbeat & Filebeats , version 7.9.0

Elastic Security SIEM
#1

Hi Team,

We are seeing that auditbeat service is using lot of memory in our VM's , we suspect that when elasticsearch cluster is down its taking more memory in auditbeat to buffer the data. Could you please let me know if we can do some update in auditbeat.yml so that it will not buffer any data in memory

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20849 root 20 0 27.4g 24.6g 38784 S 71.4 44.1 683:48.32 auditbeat

#2

Hello Team,

Kindly look into this issue. We are seeing this issue on multiple servers even if the elasticsearch cluster is UP.

One option we tried is disabled socket and process module and we could see the cpu/memory usage has considerably decreased and still monitoring. But would like to know if there is any thing more we can do to resolve this issue.

Please help

#3

(Moving to SIEM category so the Elastic team that handles Auditbeat sees this post).

#4

2nd the request. This also plays in with the ones pre packages with the ingest manager. As an idea of a cap 512Mb to 1Gb max should be all that they use with no more then 50% CPU. I had metricbeat take a few of my servers as it took 6Gb+ ram from the host and it starved out the application.

#5

You can minimize this by changing the settings for the internal queue. See the details at https://www.elastic.co/guide/en/beats/auditbeat/current/configuring-internal-queue.html. As an example:

queue.mem:
  flush.min_events: 512
  events: 1024
#6

If you are still seeing conditions when Auditbeat consumes lots of memory you can take a heap profile while it's running that can be used to identify what's consuming the space. To prepare you need to have Auditbeat running with the added CLI flag of --httpprof localhost:8080.

Then when you want to capture a heap profile you can use curl to save it.

curl -O heap1.prof http://localhost:8080/debug/pprof/heap

That file can be analyzed with go tool pprof -http=localhost:6060 heap1.prof (https://github.com/google/pprof#run-pprof-via-a-web-interface). This can help show what objects are consuming the space and where they are allocated.

#7

Thanks Andrew. As of now its fine after disabling socket and process datasets. also we have added this new config as you suggested.

queue.mem:
flush.min_events: 16
events: 32

I ill try to capture the heap profile for further analysis. Hope this heap profile capture works for filebeat as well.

#8

It will work for any Beat. Additionally it can be used to collect 30 second CPU profiles if you read from /debug/pprof/profile.