Auditbeat 7.4 crashing - too many open files

Seeing errors like these in journal:

Oct 02 18:26:50 hostname auditbeat[30239]: 2019-10-02T18:26:50.004Z        ERROR        instance/beat.go:878        
  Exiting: 1 error: 1 error: system/socket dataset setup failed: 
  unable to monitor probe p:tcp6_connect_in tcp_v6_connect 
  sock={{.P1}} laddra={{.INET_SOCK_V6_LADDR_A}}({{.P1}}){{.INET_SOCK_V6_TERM}} 
  perf_event_open: too many open files

It only seems to be impacting systems where I have iSCSI mounts though.

# uname -a
Linux hostname 3.10.0-957.12.2.el7.x86_64 #1 SMP Tue May 14 21:24:32 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

# cat /etc/redhat-release 
CentOS Linux release 7.6.1810 (Core)

# rpm -qa | grep auditbeat
**auditbeat** -7.4.0-1.x86_64

NOTE: ipv6 isn't used on these systems. Just v4.

Seems that you are hitting a limit on the number of file descriptors that auditbeat can use.

Can you check this limit (ulimit -n) and try with an increased value? (ulimit -n newlimit).

Are you running alongside the file_integrity module? Can you share your auditbeat.yml? It might be an explanation for an abnormally high descriptor usage.

Changing this setting seems to have fixed the problem:

socket.enable_ipv6: false

What you did is to reduce the number of file descriptors open by disabling a feature on the socket dataset. You may still have this problem in the future if you don't increase the maximum number of descriptors available to Auditbeat.

Yikes. Ok. TY.