Can't seem to use socket dataset on rocky linux

jhbigler · February 24, 2022, 1:41am

Auditbeat version: 7.15.2
Operating system: Rocky Linux (4.18.0-348.12.2.el8_5.x86_64)

My issue is similar to the one discussed here, but the fixes that were put in 7.10 do not work here. Commenting out the socket dataset fixes auditbeat but I also lose out on valuable information.

I can confirm:

Setting a higher socket.guess_timeout does not fix it
/proc/sys/kernel/ftrace_enabled is set to 1
/sys/kernel/debug/kprobes/enabled is also set to 1
ipv6 is disabled, both on the system and in auditbeat (this issue mentioned ipv6 but I don't think it's that)
Neither modules.d nor config reloading are being used

module config:

- module: system
  datasets:
  - host
  - login
  - package
  - process
  - socket
  - user
  socket.enable_ipv6: false

Sample of logs:

2022-02-24T01:37:01.366Z        DEBUG   [modules]       beater/metricbeat.go:151        Available modules and metricsets: Register [ModuleFactory:[system], MetricSetFactory:[auditd/auditd, file_integrity/file, system/host, system/login, system/package, system/process, system/socket, system/user]]
2022-02-24T01:37:01.367Z        WARN    [cfgwarn]       host/host.go:188        BETA: The system/host dataset is beta
2022-02-24T01:37:01.368Z        DEBUG   [system]        host/host.go:472        Restored last host information from disk.
2022-02-24T01:37:01.368Z        WARN    [cfgwarn]       login/login.go:95       BETA: The system/login dataset is beta
2022-02-24T01:37:01.368Z        DEBUG   [login] login/utmp.go:546       Restored 2 UTMP file records from disk
2022-02-24T01:37:01.368Z        DEBUG   [login] login/utmp.go:578       Restored 1 open login sessions from disk
2022-02-24T01:37:01.368Z        WARN    [cfgwarn]       package/package.go:201  BETA: The system/package dataset is beta
2022-02-24T01:37:01.369Z        DEBUG   [package]       package/package.go:232  Last state was sent at 2022-02-24 00:31:41.858757849 +0000 UTC. Next state update by 2022-02-24 12:31:41.858757849 +0000 UTC.
2022-02-24T01:37:01.369Z        DEBUG   [package]       package/package.go:242  Restored 401 packages from disk
2022-02-24T01:37:01.370Z        WARN    [cfgwarn]       process/process.go:146  BETA: The system/process dataset is beta
2022-02-24T01:37:01.370Z        DEBUG   [process]       process/process.go:183  Last state was sent at 2022-02-24 00:31:52.993867 +0000 UTC. Next state update by 2022-02-24 12:31:52.993867 +0000 UTC.
2022-02-24T01:37:01.370Z        WARN    [cfgwarn]       socket/socket_linux.go:124      BETA: The system/socket dataset is beta.
2022-02-24T01:37:01.378Z        INFO    [socket]        socket/socket_linux.go:259      Setting up system/socket for kernel 4.18.0-348.12.2.el8_5.x86_64
2022-02-24T01:37:01.379Z        DEBUG   [socket]        socket/socket_linux.go:306      IPv6 supported: false
2022-02-24T01:37:01.379Z        DEBUG   [socket]        socket/socket_linux.go:313      IPv6 enabled: false
2022-02-24T01:37:01.514Z        DEBUG   [socket]        socket/socket_linux.go:374      Selected kernel function __ip_local_out for IP_LOCAL_OUT
2022-02-24T01:37:01.514Z        DEBUG   [socket]        socket/socket_linux.go:374      Selected kernel function __skb_recv_udp for RECV_UDP_DATAGRAM
2022-02-24T01:37:01.514Z        DEBUG   [socket]        socket/socket_linux.go:374      Selected kernel function __x64_sys_execve for SYS_EXECVE
2022-02-24T01:37:01.514Z        DEBUG   [socket]        socket/socket_linux.go:374      Selected kernel function __x64_sys_gettimeofday for SYS_GETTIMEOFDAY
2022-02-24T01:37:01.514Z        DEBUG   [socket]        socket/socket_linux.go:374      Selected kernel function __x64_sys_newuname for SYS_UNAME
2022-02-24T01:37:01.520Z        INFO    [socket]        guess/guess.go:258      Running 17 guesses ...
2022-02-24T01:37:01.520Z        DEBUG   [socket]        guess/guess.go:270      Guess guess_inet_sock_ipv6 skipped.
2022-02-24T01:37:16.558Z        WARN    [cfgwarn]       user/user.go:232        BETA: The system/user dataset is beta
2022-02-24T01:37:16.559Z        DEBUG   [user]  user/user.go:272        Last state was sent at 2022-02-24 00:31:42.295524292 +0000 UTC. Next state update by 2022-02-24 12:31:42.295524292 +0000 UTC.
2022-02-24T01:37:16.559Z        DEBUG   [user]  user/user.go:282        Restored 24 users from disk
2022-02-24T01:37:16.559Z        INFO    instance/beat.go:442    auditbeat stopped.
2022-02-24T01:37:16.559Z        ERROR   instance/beat.go:989    Exiting: 1 error: system/socket dataset setup failed: unable to guess one or more required parameters: guess_sockaddr_in failed: timeout while waiting for trigger to complete
Exiting: 1 error: system/socket dataset setup failed: unable to guess one or more required parameters: guess_sockaddr_in failed: timeout while waiting for trigger to complete

The parameter it fails on can vary - sometimes it's guess_udp_sendmsg, sometimes it's others.

Any ideas of what could be going wrong? Weirdly enough, this did work fine at one point, but now it crashes.

Many thanks!

TiagoQueiroz · February 24, 2022, 12:40pm

Hi @jhbigler I did not manage to reproduce this issue with Auditbeat 7.15.2. I'm running Rocky Linux on a VM, but with the same kernel

[root@SysNet4Admin auditbeat-7.15.2-linux-x86_64]# ./auditbeat version
auditbeat version 7.15.2 (amd64), libbeat 7.15.2 [fd322dad6ceafec40c84df4d2a0694ea357d16cc built 2021-11-04 14:16:53 +0000 UTC]
[root@SysNet4Admin auditbeat-7.15.2-linux-x86_64]# uname -a
Linux SysNet4Admin 4.18.0-348.12.2.el8_5.x86_64 #1 SMP Wed Jan 19 17:53:40 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
[root@SysNet4Admin auditbeat-7.15.2-linux-x86_64]#

Does it fail right at the start up or does it take a while?

How did you install Auditbeat? Did yo download directly from our download page?

Could you post here you full configuration (please, redact the credentials and other sensitive information).

jhbigler · February 24, 2022, 6:51pm

Hello,

I used the RPM from elastic's repo. Yes, it fails at start once it times out its 'guesses' (see logs).

auditbeat.modules:
- module: system
  datasets:
  - host
  - login
  - package
  - process
  - socket
  - user
- module: file_integrity
  paths:
  - /bin
  - /usr/bin
  - /sbin
  - /usr/bin
  - /etc/metricbeat/metricbeat.yml
  - /etc/auditbeat/auditbeat.yml
  - /etc/journalbeat/journalbeat.yml
  - /etc/puppetlabs/puppet/puppet.conf
  - /etc/logstash/conf.d
  - /etc/haproxy
  - /etc/ssh/
  - /etc/firewalld/
  - /usr/lib/systemd/
  exclude_files:
  - '~$'
  - '.*/data/.*'
  - '\.json$'
  - '\.log$'
  - '\.lock$'
  scan_at_start: true
  max_file_size: 100 MiB
  hash_types: [sha1]
  recursive: true
- module: auditd
  audit_rules: |
    -a exit,always -S unlink -S rmdir
    -w /etc/group -p wa
    -w /etc/passwd -p wa
    -w /etc/shadow -p wa
    -w /etc/sudoers -p wa
    -a exit,always -F euid=0 -S execve

fields:
  isdname: "..."
fields_under_root: true
tags: ["project", "audit"]

processors:
- drop_fields:
    fields: ["@metadata","log","ecs","input","agent","host","hostname"]
    ignore_missing: true
- add_id:
    target_field: "event_id"
- fingerprint:
    fields: ["isdname","event_id"]
    target_field: "@metadata._id"
    method: "sha1"
- drop_fields:
    fields: ["event_id"]
    ignore_missing: true

output.kafka:
  hosts:
  - "..."
  topic: "project-audit"
  partition.round_robin:
    reachable_only: true

  keep_alive: '60s'

  ssl:
    certificate: "/opt/isd/ssl/kafka.pem"
    certificate_authorities: ["/opt/isd/ssl/ca.pem"]
    key: "/opt/isd/ssl/kafka.key"
    verification_mode: "certificate"

queue.disk:
  path: /opt/isd/data/auditbeat
  max_size: 30GB
  segment_size: 1GB

TiagoQueiroz · March 2, 2022, 5:16pm

@jhbigler I haven't managed to reproduce it yet

I'm trying to get help from someone more experienced with auditbeat.

adrisr · March 2, 2022, 5:24pm

Hi,

Is the reason of the error always "timeout while waiting for trigger to complete"? Or it gives different reasons?

Can you share a few different errors ?

adrisr · March 2, 2022, 5:54pm

By looking at the code, I see a reason why the trigger for both guess_sockaddr_in and guess_udp_sendmsg can timeout.

Those two probes, and maybe others, require the loopback interface to be assigned the network 127.0.0.1/8. They will try to communicate using addresses like 127.123.123.123 (actually a random address). Maybe a change in your system is preventing such communication by having the loopback associated to 127.0.0.1/32 or 127.0.0.1/24.

Can you check the output of any of these commands:

ip a ls dev lo
should return:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever

ip route ls table local dev lo

broadcast 127.0.0.0 proto kernel scope link src 127.0.0.1
local 127.0.0.0/8 proto kernel scope host src 127.0.0.1
local 127.0.0.1 proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 proto kernel scope link src 127.0.0.1

And also check communication with the following commands in different terminals:

(server)
nc -vln -s 127.123.123.123 -p 8888

(client)
echo hello | nc -v 127.123.123.123 8888

If communication if possible the server should output:

listening on [127.123.123.123] 8888 ...
connect to [127.123.123.123] from (UNKNOWN) [127.0.0.1] 55994
hello

jhbigler · March 2, 2022, 8:57pm

I've discovered that turning off firewalld fixes the issue on auditbeat. Obviously, we are not willing to simply turn off firewalld on all of these machines... I will continue to dig down on if it's a specific rule that's breaking the socket dataset (we use direct rules to severely limit outgoing communication, in addition to rich rules to limit inbound packets) or if it's simply the presence of firewalld. While I do this, are there any network requirements for the socket dataset to work? Other than, obviously, what is needed for any auditbeat outputs.

jhbigler · March 2, 2022, 9:00pm

Also to answer @adrisr 's questions, all of those tests check out on the machine. Firewalld seems to be the culprit (see above)

system · March 30, 2022, 11:00pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.