Auditbeat 7.4 crashing - too many open files

[ethrbunny]

Seeing errors like these in journal:

Oct 02 18:26:50 hostname auditbeat[30239]: 2019-10-02T18:26:50.004Z        ERROR        instance/beat.go:878        
  Exiting: 1 error: 1 error: system/socket dataset setup failed: 
  unable to monitor probe p:tcp6_connect_in tcp_v6_connect 
  sock={{.P1}} laddra={{.INET_SOCK_V6_LADDR_A}}({{.P1}}){{.INET_SOCK_V6_TERM}} 
  laddrb={{.INET_SOCK_V6_LADDR_B}}({{.P1}}){{.INET_SOCK_V6_TERM}} 
  lport=+{{.INET_SOCK_LPORT}}({{.P1}}):u16 
  af=+{{.SOCKADDR_IN6_AF}}({{.P2}}):u16 
  addra=+{{.SOCKADDR_IN6_ADDRA}}({{.P2}}):u64 
  addrb=+{{.SOCKADDR_IN6_ADDRB}}({{.P2}}):u64 
  port=+{{.SOCKADDR_IN6_PORT}}({{.P2}}):u16: 
  perf_event_open: too many open files

It only seems to be impacting systems where I have iSCSI mounts though.

# uname -a
Linux hostname 3.10.0-957.12.2.el7.x86_64 #1 SMP Tue May 14 21:24:32 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

# cat /etc/redhat-release 
CentOS Linux release 7.6.1810 (Core)

# rpm -qa | grep auditbeat
**auditbeat** -7.4.0-1.x86_64

NOTE: ipv6 isn't used on these systems. Just v4.

[adrisr (Adrian Serrano)]

@ethrbunny

  perf_event_open: too many open files

Seems that you are hitting a limit on the number of file descriptors that auditbeat can use.

Can you check this limit (ulimit -n) and try with an increased value? (ulimit -n newlimit).

Are you running alongside the file_integrity module? Can you share your auditbeat.yml? It might be an explanation for an abnormally high descriptor usage.

[ethrbunny]

Changing this setting seems to have fixed the problem:

socket.enable_ipv6: false

[adrisr (Adrian Serrano)]

What you did is to reduce the number of file descriptors open by disabling a feature on the socket dataset. You may still have this problem in the future if you don't increase the maximum number of descriptors available to Auditbeat.

[ethrbunny]

Yikes. Ok. TY.