Auditbeat process memory grows every day

[v1k1ng0 (Oliver)]

Hi,

not sure if happening in all of my servers (vmware virtual servers), but I have 2 servers with antivirus installed (trend micro deep security) and auditbeat installed happening that auditbeat process memory grows 2% every day.

I've read some posts about memory leak and I've modified config file (/etc/auditbeat/auditbeat.yml) adding next lines:

Internal queue configuration for buffering events to be published.

queue:
mem:
events: 1024
flush.min_events: 512
flush.timeout: 1s

And I've added more lines in processors section for adding drop_event when equals process.excecutable (and here I've put path from different process executables of deep security). Can someone help me please? Thanks beforehand.

Cheers...

[willemdh]

I've seen similar behaviour. What modules are enabled and how are they configured? Can you paste the sanited content of your auditbeat.yml ?

[v1k1ng0 (Oliver)]

Thanks for replying. Here it is:

auditbeat.modules:

• module: auditd
  audit_rule_files: [ '${path.config}/audit.rules.d/*.conf' ]
  audit_rules: |

• module: file_integrity
  paths:

  • /bin
  • /usr/bin
  • /sbin
  • /usr/sbin
  • /etc

• module: system
  datasets:

  • package

  period: 2m

• module: system
  datasets:

  • host
  • login
  • process
  • socket
  • user

  state.period: 12h

  user.detect_password_changes: true

  login.wtmp_file_pattern: /var/log/wtmp*
  login.btmp_file_pattern: /var/log/btmp*

setup.template.settings:
index.number_of_shards: 1

setup.ilm.enabled: true
setup.ilm.overwrite: true
setup.ilm.policy_name: 5-years
setup.ilm.check_exists: false

queue:
mem:
events: 1024
flush.min_events: 512
flush.timeout: 1s

setup.kibana:

host: "xxx://xxx:xxx"
proxy_disable: true

output.elasticsearch:
hosts: ["xxx://xxx:xxx"]
api_key: xxx:xxx
proxy_disable: true

processors:

• add_host_metadata: ~
• drop_event:
  when:
  equals:
  process.executable: '/usr/sbin/ntpd'
• drop_event:
  when:
  equals:
  user.name: 'xxx'
• drop_event:
  when:
  equals:
  process.executable: '/opt/ds_agent/ds_agent'
• drop_event:
  when:
  equals:
  process.executable: '/opt/ds_agent/ds_asm'
• drop_event:
  when:
  equals:
  process.executable: '/opt/ds_agent/dsa-connect'
• drop_event:
  when:
  equals:
  process.executable: '/opt/ds_agent/nuagent/ds_nuagent'

[willemdh]

Ok, now disable each module but one and try to find out what module is causing the memory leak. I think it's the auditd or the file_integrity. Also please post your auditd config if possible. (had to clean up some noisy auditd entries on some of my servers in the past)

[v1k1ng0 (Oliver)]

@willemdh

Ok, now disable each module but one and try to find out what module is causing the memory leak. I think it's the auditd or the file_integrity. Also please post your auditd config if possible. (had to clean up some noisy auditd entries on some of my servers in the past)

Thanks.
What's the config file of auditd? You can see auditd module configuration in auditbeat.yml in the before post. It calls to /audit.rules.d/*.conf and within here I have a file named sample-rules.conf.disabled, so I understand the it is not being used. Thanks beforehand.

[willemdh]

Weird that your auditd is empty. Then why put it in your config? Please check Auditd Module | Auditbeat Reference [8.10] | Elastic and list the auditd rules. If empty, then disable the module.

auditbeat show auditd-rules

To understand what module might be leaking memory, I suggest you disable all but one and check your process memory. Example:

- module: auditd
  enabled: false

Does it stop growing eventually? How much max memory have you seen for the process?

Grtz

[v1k1ng0 (Oliver)]

Thanks for replying me.

Auditbeat was set up by a colleague, I'm looking at the memory leak issue. "Show auditd-rules" command shows empty rules, so I will test to disable it.

I'm sorry for taking some days for replying you, but I have to wait some days for seeing if process memory stops of growing..

When I restart the process, it starts with 100MB more or less and I've seen that it grows to 2,XGB some days after.. so I have to restart it again. Thanks beforehand.

Cheers...