Auditbeat process memory grows every day

Elastic Stack Beats
1

Hi,

not sure if happening in all of my servers (vmware virtual servers), but I have 2 servers with antivirus installed (trend micro deep security) and auditbeat installed happening that auditbeat process memory grows 2% every day.

I've read some posts about memory leak and I've modified config file (/etc/auditbeat/auditbeat.yml) adding next lines:

Internal queue configuration for buffering events to be published.

queue:
mem:
events: 1024
flush.min_events: 512
flush.timeout: 1s

And I've added more lines in processors section for adding drop_event when equals process.excecutable (and here I've put path from different process executables of deep security). Can someone help me please? Thanks beforehand.

Cheers...

2

I've seen similar behaviour. What modules are enabled and how are they configured? Can you paste the sanited content of your auditbeat.yml ?

3

Thanks for replying. Here it is:

auditbeat.modules:

  • module: auditd
    audit_rule_files: [ '${path.config}/audit.rules.d/*.conf' ]
    audit_rules: |

  • module: file_integrity
    paths:

    • /bin
    • /usr/bin
    • /sbin
    • /usr/sbin
    • /etc

  • module: system
    datasets:

    • package

    period: 2m

  • module: system
    datasets:

    • host
    • login
    • process
    • socket
    • user

    state.period: 12h

    user.detect_password_changes: true

    login.wtmp_file_pattern: /var/log/wtmp*
    login.btmp_file_pattern: /var/log/btmp*

setup.template.settings:
index.number_of_shards: 1

setup.ilm.enabled: true
setup.ilm.overwrite: true
setup.ilm.policy_name: 5-years
setup.ilm.check_exists: false

queue:
mem:
events: 1024
flush.min_events: 512
flush.timeout: 1s

setup.kibana:

host: "xxx://xxx:xxx"
proxy_disable: true

output.elasticsearch:
hosts: ["xxx://xxx:xxx"]
api_key: xxx:xxx
proxy_disable: true

processors:

  • add_host_metadata: ~
  • drop_event:
    when:
    equals:
    process.executable: '/usr/sbin/ntpd'
  • drop_event:
    when:
    equals:
    user.name: 'xxx'
  • drop_event:
    when:
    equals:
    process.executable: '/opt/ds_agent/ds_agent'
  • drop_event:
    when:
    equals:
    process.executable: '/opt/ds_agent/ds_asm'
  • drop_event:
    when:
    equals:
    process.executable: '/opt/ds_agent/dsa-connect'
  • drop_event:
    when:
    equals:
    process.executable: '/opt/ds_agent/nuagent/ds_nuagent'
4

Ok, now disable each module but one and try to find out what module is causing the memory leak. I think it's the auditd or the file_integrity. Also please post your auditd config if possible. (had to clean up some noisy auditd entries on some of my servers in the past)

5

Thanks.
What's the config file of auditd? You can see auditd module configuration in auditbeat.yml in the before post. It calls to /audit.rules.d/*.conf and within here I have a file named sample-rules.conf.disabled, so I understand the it is not being used. Thanks beforehand.

6

Weird that your auditd is empty. Then why put it in your config? Please check Auditd Module | Auditbeat Reference [8.10] | Elastic and list the auditd rules. If empty, then disable the module.

auditbeat show auditd-rules

To understand what module might be leaking memory, I suggest you disable all but one and check your process memory. Example:

- module: auditd
  enabled: false

Does it stop growing eventually? How much max memory have you seen for the process?

Grtz

7

Thanks for replying me.

Auditbeat was set up by a colleague, I'm looking at the memory leak issue. "Show auditd-rules" command shows empty rules, so I will test to disable it.

I'm sorry for taking some days for replying you, but I have to wait some days for seeing if process memory stops of growing..

When I restart the process, it starts with 100MB more or less and I've seen that it grows to 2,XGB some days after.. so I have to restart it again. Thanks beforehand.

Cheers...

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.