Auditbeat 7.4 crashing - too many open files

Seeing errors like these in journal:

Oct 02 18:26:50 hostname auditbeat[30239]: 2019-10-02T18:26:50.004Z        ERROR        instance/beat.go:878        
  Exiting: 1 error: 1 error: system/socket dataset setup failed: 
  unable to monitor probe p:tcp6_connect_in tcp_v6_connect 
  sock={{.P1}} laddra={{.INET_SOCK_V6_LADDR_A}}({{.P1}}){{.INET_SOCK_V6_TERM}} 
  perf_event_open: too many open files

It only seems to be impacting systems where I have iSCSI mounts though.

# uname -a
Linux hostname 3.10.0-957.12.2.el7.x86_64 #1 SMP Tue May 14 21:24:32 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

# cat /etc/redhat-release 
CentOS Linux release 7.6.1810 (Core)

# rpm -qa | grep auditbeat
**auditbeat** -7.4.0-1.x86_64

NOTE: ipv6 isn't used on these systems. Just v4.

Seems that you are hitting a limit on the number of file descriptors that auditbeat can use.

Can you check this limit (ulimit -n) and try with an increased value? (ulimit -n newlimit).

Are you running alongside the file_integrity module? Can you share your auditbeat.yml? It might be an explanation for an abnormally high descriptor usage.

Changing this setting seems to have fixed the problem:

socket.enable_ipv6: false

What you did is to reduce the number of file descriptors open by disabling a feature on the socket dataset. You may still have this problem in the future if you don't increase the maximum number of descriptors available to Auditbeat.

Yikes. Ok. TY.

This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.