# Auditbeat memory leak

**URL:** <https://discuss.elastic.co/t/auditbeat-memory-leak/259137>\
**Category:** Beats\
**Tags:** auditbeat\
**Created:** [December 18, 2020, 7:53pm UTC](https://discuss.elastic.co/t/auditbeat-memory-leak/259137 "2020-12-18T19:53:41Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![BenB196](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benb196/32/83401_2.png) [@BenB196](https://discuss.elastic.co/u/BenB196)\
**Post date:** [December 18, 2020, 7:53pm UTC](https://discuss.elastic.co/t/auditbeat-memory-leak/259137/1 "2020-12-18T19:53:41Z")

</div>

Hi All,

I was wondering if anyone else has noticed a memory leak with the last few version of Auditbeat?

Currently running 7.10.0 (and had the issue in 7.8 and 7.9 as well) and I am seeing Auditbeat constantly being OOM killed every day or so. I'm only seeing it on relatively high traffic DNS servers. The DNS servers in question receive between 400-100 queries per second which doesn't seem like anything that should be causing Auditbeat to consume so much resources.

Here is the config I'm running:

```auto
#========================== Modules configuration =============================
auditbeat.modules:

- module: auditd
  resolve_ids: true
  failure_mode: silent
  backlog_limit: 8192
  rate_limit: 0
  include_raw_message: false
  include_warnings: false
  backpressure_stratgey: auto
  # Load audit rules from separate files. Same format as audit.rules(7).
  audit_rule_files: ['${path.config}/audit.rules.d/*.conf']
  audit_rules: |
    ## Define audit rules here.
    ## Create file watches (-w) or syscall audits (-a or -A). Uncomment these
    ## examples or add your own rules.

    ## If you are on a 64 bit platform, everything should be running
    ## in 64 bit mode. This rule will detect any use of the 32 bit syscalls
    ## because this might be a sign of someone exploiting a hole in the 32
    ## bit API.
    -a always,exit -F arch=b32 -S all -F key=32bit-abi

    ## Executions.
    -a always,exit -F arch=b64 -S execve,execveat -k exec

    ## External access (warning: these can be expensive to audit).
    ##-a always,exit -F arch=b64 -S accept,bind,connect -F key=external-access
    ##-a always,exit -F arch=b64 -S accept,bind -F key=external-access

    ## Identity changes.
    -w /etc/group -p wa -k identity
    -w /etc/passwd -p wa -k identity
    -w /etc/gshadow -p wa -k identity
    -w /etc/shadow -p wa -k identity

    ## Unauthorized access attempts.
    -a always,exit -F arch=b32 -S open,creat,truncate,ftruncate,openat,open_by_handle_at -F exit=-EACCES -k access
    -a always,exit -F arch=b32 -S open,creat,truncate,ftruncate,openat,open_by_handle_at -F exit=-EPERM -k access
    -a always,exit -F arch=b64 -S open,creat,truncate,ftruncate,openat,open_by_handle_at -F exit=-EACCES -k access
    -a always,exit -F arch=b64 -S open,creat,truncate,ftruncate,openat,open_by_handle_at -F exit=-EPERM -k access

- module: file_integrity
  paths:
  - /bin
  - /usr/bin
  - /sbin
  - /usr/sbin
  - /etc
  - /root
  - /usr/local/bin
  - /home
  exclude_files:
  - '(?i)\.sw[nop]$'
  - '~$'
  - '/\.git($|/)'
  - '\.rrd$'
  include_files: []
  scan_at_start: true
  scan_rate_per_sec: 50 MiB
  max_file_size: 100 MiB
  hash_types: [md5,sha256]
  recursive: true

- module: system
  datasets:
    - host # General host information, e.g. uptime, IPs
    - login # User logins, logouts, and system boots.
    - package # Installed, updated, and removed packages
    - process # Started and stopped processes
    - socket # Opened and closed sockets
    - user # User information

  # How often datasets send state updates with the
  # current state of the system (e.g. all currently
  # running processes, all open sockets).
  state.period: 12h

  # Enabled by default. Auditbeat will read password fields in
  # /etc/passwd and /etc/shadow and store a hash locally to
  # detect any changes.
  user.detect_password_changes: true

  # File patterns of the login record files.
  login.wtmp_file_pattern: /var/log/wtmp*
  login.btmp_file_pattern: /var/log/btmp*

#================================ Outputs =====================================

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["<snipped>","<snipped>","<snipped>"]
  loadbalance: true

#================================ Processors =====================================

# Configure processors to enhance or manipulate events generated by the beat.

processors:
  - add_host_metadata: ~
  - add_tags:
      tags: [auditbeat]
  - dns:
      type: reverse
      fields:
        server.ip: server.hostname
        client.ip: client.hostname
        source.ip: source.hostname
        destination.ip: destination.hostname
      nameservers: ['<snipped>','<snipped>','<snipped>']
      tag_on_failure: [_dns_reverse_lookup_failed]
# - add_cloud_metadata: ~
# - add_docker_metadata: ~

#================================ Logging =====================================

logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/auditbeat
  name: auditbeat
  keepfiles: 2
  permissions: 0600
  rotateeverybytes: 5242880

#============================== X-Pack Monitoring ===============================

monitoring.enabled: true
monitoring.elasticsearch:
  hosts: ["<snipped>","<snipped>","<snipped>"]
  protocol: "https"
  username: "<snipped>"
  password: "<snipped>"
  ssl.enabled: true
  ssl.verification_mode: full
  ssl.certificate_authorities: ["<snipped>"]
monitoring.cluster_uuid: "<snipped>"

```

The DNS servers are CentOS 8:

```auto
# cat /etc/os-release
NAME="CentOS Linux"
VERSION="8 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="8"
PLATFORM_ID="platform:el8"
PRETTY_NAME="CentOS Linux 8 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:8"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-8"
CENTOS_MANTISBT_PROJECT_VERSION="8"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="8"

```

Auditbeat will slowly eat RAM over time, until Linux will OOM kill it, and the process requires a restart.

Below is a screen shot of it occurring:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/7/b76fef99d8e9384d2d1484cd48e15286e50f3c2a.png)

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [December 18, 2020, 8:24pm UTC](https://discuss.elastic.co/t/auditbeat-memory-leak/259137/2 "2020-12-18T20:24:35Z")

</div>

If you can capture a memory profile when it gets high that will probably help figure out what is leaking. If you add `--httpprof=localhost:5060` to the service's arguments then this will allow you to make a `curl http://localhost:5060/debug/pprof/heap > heap.bin` request to collect a profile that we can analyze with `go tool pprof -web hap.bin`.

After you collect the profile I think I would try disabling the DNS correlation for the system.socket dataset with `socket.dns.enabled: false` in the system module settings. [https://www.elastic.co/guide/en/beats/auditbeat/current/auditbeat-dataset-system-socket.html](https://www.elastic.co/guide/en/beats/auditbeat/current/auditbeat-dataset-system-socket.html)

```auto
- module: system
  datasets:
    - socket
    - ...
  socket.dns.enabled: false

```

---

<div class="post-metadata">

**Author:** ![BenB196](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benb196/32/83401_2.png) [@BenB196](https://discuss.elastic.co/u/BenB196)\
**Post date:** [December 18, 2020, 8:52pm UTC](https://discuss.elastic.co/t/auditbeat-memory-leak/259137/3 "2020-12-18T20:52:57Z")

</div>

Thanks, I've added the argument to one of the servers and will wait for the memory to start being used again.

---

<div class="post-metadata">

**Author:** ![BenB196](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benb196/32/83401_2.png) [@BenB196](https://discuss.elastic.co/u/BenB196)\
**Post date:** [December 21, 2020, 4:52pm UTC](https://discuss.elastic.co/t/auditbeat-memory-leak/259137/4 "2020-12-21T16:52:40Z")

</div>

@andrewkroh I wasn't able to upload the .bin file here. I've attached a screenshot of the graph instead.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/3/d/3d69449f84ac956013bf9af96b1585b59795bfa7.png)

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [December 21, 2020, 8:07pm UTC](https://discuss.elastic.co/t/auditbeat-memory-leak/259137/5 "2020-12-21T20:07:56Z")

</div>

I looked over the source code. It may not be a leak, but just a large cache that is constantly full of new DNS transactions. After 30s any DNS metadata is removed from the cache. It's probably best to turn the feature off for a DNS server.

It captures packets with `udp src port 53` that have A or AAAA dns answers. Then when the host makes a "connection" to one of those IPs it enriches that connection event with the hostname from the DNS packet. For a DNS server this will capture a lot more packets.

---

<div class="post-metadata">

**Author:** ![BenB196](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benb196/32/83401_2.png) [@BenB196](https://discuss.elastic.co/u/BenB196)\
**Post date:** [December 23, 2020, 1:28pm UTC](https://discuss.elastic.co/t/auditbeat-memory-leak/259137/6 "2020-12-23T13:28:55Z")

</div>

@andrewkroh After letting Auditbeat run for a bit with `socket.dns.enabled: false` , I am no longer seeing the memory issues. Thanks for the help. For my future reference/understanding, is this something that should be done for all DNS server?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 20, 2021, 3:28pm UTC](https://discuss.elastic.co/t/auditbeat-memory-leak/259137/7 "2021-01-20T15:28:59Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.