# Can't seem to use socket dataset on rocky linux **URL:** \ **Category:** Beats\ **Tags:** auditbeat\ **Created:** [February 24, 2022, 1:41am UTC](https://discuss.elastic.co/t/cant-seem-to-use-socket-dataset-on-rocky-linux/298102 "2022-02-24T01:41:08Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![jhbigler](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jhbigler/32/99133_2.png) [@jhbigler](https://discuss.elastic.co/u/jhbigler)\ **Post date:** [February 24, 2022, 1:41am UTC](https://discuss.elastic.co/t/cant-seem-to-use-socket-dataset-on-rocky-linux/298102/1 "2022-02-24T01:41:08Z")
Auditbeat version: 7.15.2 Operating system: Rocky Linux (4.18.0-348.12.2.el8\_5.x86\_64) My issue is similar to the one discussed [here](https://github.com/elastic/beats/issues/18755), but the fixes that were put in 7.10 do not work here. Commenting out the socket dataset fixes auditbeat but I also lose out on valuable information. I can confirm: - Setting a higher socket.guess\_timeout does not fix it - /proc/sys/kernel/ftrace\_enabled is set to 1 - /sys/kernel/debug/kprobes/enabled is also set to 1 - ipv6 is disabled, both on the system and in auditbeat ([this issue](https://discuss.elastic.co/t/system-socket-module-stops-auditbeat-7-4-from-starting-ipv6-detection/201852) mentioned ipv6 but I don't think it's that) - Neither modules.d nor config reloading are being used module config: ```auto - module: system datasets: - host - login - package - process - socket - user socket.enable_ipv6: false ``` Sample of logs: ```auto 2022-02-24T01:37:01.366Z DEBUG [modules] beater/metricbeat.go:151 Available modules and metricsets: Register [ModuleFactory:[system], MetricSetFactory:[auditd/auditd, file_integrity/file, system/host, system/login, system/package, system/process, system/socket, system/user]] 2022-02-24T01:37:01.367Z WARN [cfgwarn] host/host.go:188 BETA: The system/host dataset is beta 2022-02-24T01:37:01.368Z DEBUG [system] host/host.go:472 Restored last host information from disk. 2022-02-24T01:37:01.368Z WARN [cfgwarn] login/login.go:95 BETA: The system/login dataset is beta 2022-02-24T01:37:01.368Z DEBUG [login] login/utmp.go:546 Restored 2 UTMP file records from disk 2022-02-24T01:37:01.368Z DEBUG [login] login/utmp.go:578 Restored 1 open login sessions from disk 2022-02-24T01:37:01.368Z WARN [cfgwarn] package/package.go:201 BETA: The system/package dataset is beta 2022-02-24T01:37:01.369Z DEBUG [package] package/package.go:232 Last state was sent at 2022-02-24 00:31:41.858757849 +0000 UTC. Next state update by 2022-02-24 12:31:41.858757849 +0000 UTC. 2022-02-24T01:37:01.369Z DEBUG [package] package/package.go:242 Restored 401 packages from disk 2022-02-24T01:37:01.370Z WARN [cfgwarn] process/process.go:146 BETA: The system/process dataset is beta 2022-02-24T01:37:01.370Z DEBUG [process] process/process.go:183 Last state was sent at 2022-02-24 00:31:52.993867 +0000 UTC. Next state update by 2022-02-24 12:31:52.993867 +0000 UTC. 2022-02-24T01:37:01.370Z WARN [cfgwarn] socket/socket_linux.go:124 BETA: The system/socket dataset is beta. 2022-02-24T01:37:01.378Z INFO [socket] socket/socket_linux.go:259 Setting up system/socket for kernel 4.18.0-348.12.2.el8_5.x86_64 2022-02-24T01:37:01.379Z DEBUG [socket] socket/socket_linux.go:306 IPv6 supported: false 2022-02-24T01:37:01.379Z DEBUG [socket] socket/socket_linux.go:313 IPv6 enabled: false 2022-02-24T01:37:01.514Z DEBUG [socket] socket/socket_linux.go:374 Selected kernel function __ip_local_out for IP_LOCAL_OUT 2022-02-24T01:37:01.514Z DEBUG [socket] socket/socket_linux.go:374 Selected kernel function __skb_recv_udp for RECV_UDP_DATAGRAM 2022-02-24T01:37:01.514Z DEBUG [socket] socket/socket_linux.go:374 Selected kernel function __x64_sys_execve for SYS_EXECVE 2022-02-24T01:37:01.514Z DEBUG [socket] socket/socket_linux.go:374 Selected kernel function __x64_sys_gettimeofday for SYS_GETTIMEOFDAY 2022-02-24T01:37:01.514Z DEBUG [socket] socket/socket_linux.go:374 Selected kernel function __x64_sys_newuname for SYS_UNAME 2022-02-24T01:37:01.520Z INFO [socket] guess/guess.go:258 Running 17 guesses ... 2022-02-24T01:37:01.520Z DEBUG [socket] guess/guess.go:270 Guess guess_inet_sock_ipv6 skipped. 2022-02-24T01:37:16.558Z WARN [cfgwarn] user/user.go:232 BETA: The system/user dataset is beta 2022-02-24T01:37:16.559Z DEBUG [user] user/user.go:272 Last state was sent at 2022-02-24 00:31:42.295524292 +0000 UTC. Next state update by 2022-02-24 12:31:42.295524292 +0000 UTC. 2022-02-24T01:37:16.559Z DEBUG [user] user/user.go:282 Restored 24 users from disk 2022-02-24T01:37:16.559Z INFO instance/beat.go:442 auditbeat stopped. 2022-02-24T01:37:16.559Z ERROR instance/beat.go:989 Exiting: 1 error: system/socket dataset setup failed: unable to guess one or more required parameters: guess_sockaddr_in failed: timeout while waiting for trigger to complete Exiting: 1 error: system/socket dataset setup failed: unable to guess one or more required parameters: guess_sockaddr_in failed: timeout while waiting for trigger to complete ``` The parameter it fails on can vary - sometimes it's guess\_udp\_sendmsg, sometimes it's others. Any ideas of what could be going wrong? Weirdly enough, this did work fine at one point, but now it crashes. Many thanks! ---
**Author:** ![TiagoQueiroz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tiagoqueiroz/32/107061_2.png) [@TiagoQueiroz](https://discuss.elastic.co/u/TiagoQueiroz)\ **Post date:** [February 24, 2022, 12:40pm UTC](https://discuss.elastic.co/t/cant-seem-to-use-socket-dataset-on-rocky-linux/298102/2 "2022-02-24T12:40:45Z")
Hi @jhbigler I did not manage to reproduce this issue with Auditbeat 7.15.2. I'm running Rocky Linux on a VM, but with the same kernel ```nohighlight [root@SysNet4Admin auditbeat-7.15.2-linux-x86_64]# ./auditbeat version auditbeat version 7.15.2 (amd64), libbeat 7.15.2 [fd322dad6ceafec40c84df4d2a0694ea357d16cc built 2021-11-04 14:16:53 +0000 UTC] [root@SysNet4Admin auditbeat-7.15.2-linux-x86_64]# uname -a Linux SysNet4Admin 4.18.0-348.12.2.el8_5.x86_64 #1 SMP Wed Jan 19 17:53:40 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux [root@SysNet4Admin auditbeat-7.15.2-linux-x86_64]# ``` Does it fail right at the start up or does it take a while? How did you install Auditbeat? Did yo download directly from our download page? Could you post here you full configuration (please, redact the credentials and other sensitive information). ---
**Author:** ![jhbigler](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jhbigler/32/99133_2.png) [@jhbigler](https://discuss.elastic.co/u/jhbigler)\ **Post date:** [February 24, 2022, 6:51pm UTC](https://discuss.elastic.co/t/cant-seem-to-use-socket-dataset-on-rocky-linux/298102/3 "2022-02-24T18:51:33Z")
Hello, I used the RPM from elastic's repo. Yes, it fails at start once it times out its 'guesses' (see logs). ```auto auditbeat.modules: - module: system datasets: - host - login - package - process - socket - user - module: file_integrity paths: - /bin - /usr/bin - /sbin - /usr/bin - /etc/metricbeat/metricbeat.yml - /etc/auditbeat/auditbeat.yml - /etc/journalbeat/journalbeat.yml - /etc/puppetlabs/puppet/puppet.conf - /etc/logstash/conf.d - /etc/haproxy - /etc/ssh/ - /etc/firewalld/ - /usr/lib/systemd/ exclude_files: - '~$' - '.*/data/.*' - '\.json$' - '\.log$' - '\.lock$' scan_at_start: true max_file_size: 100 MiB hash_types: [sha1] recursive: true - module: auditd audit_rules: | -a exit,always -S unlink -S rmdir -w /etc/group -p wa -w /etc/passwd -p wa -w /etc/shadow -p wa -w /etc/sudoers -p wa -a exit,always -F euid=0 -S execve fields: isdname: "..." fields_under_root: true tags: ["project", "audit"] processors: - drop_fields: fields: ["@metadata","log","ecs","input","agent","host","hostname"] ignore_missing: true - add_id: target_field: "event_id" - fingerprint: fields: ["isdname","event_id"] target_field: "@metadata._id" method: "sha1" - drop_fields: fields: ["event_id"] ignore_missing: true output.kafka: hosts: - "..." topic: "project-audit" partition.round_robin: reachable_only: true keep_alive: '60s' ssl: certificate: "/opt/isd/ssl/kafka.pem" certificate_authorities: ["/opt/isd/ssl/ca.pem"] key: "/opt/isd/ssl/kafka.key" verification_mode: "certificate" queue.disk: path: /opt/isd/data/auditbeat max_size: 30GB segment_size: 1GB ``` ---
**Author:** ![TiagoQueiroz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tiagoqueiroz/32/107061_2.png) [@TiagoQueiroz](https://discuss.elastic.co/u/TiagoQueiroz)\ **Post date:** [March 2, 2022, 5:16pm UTC](https://discuss.elastic.co/t/cant-seem-to-use-socket-dataset-on-rocky-linux/298102/4 "2022-03-02T17:16:38Z")
@jhbigler I haven't managed to reproduce it yet 😕 I'm trying to get help from someone more experienced with auditbeat. ---
**Author:** ![adrisr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adrisr/32/25423_2.png) [@adrisr](https://discuss.elastic.co/u/adrisr)\ **Post date:** [March 2, 2022, 5:24pm UTC](https://discuss.elastic.co/t/cant-seem-to-use-socket-dataset-on-rocky-linux/298102/5 "2022-03-02T17:24:43Z")
Hi, Is the reason of the error always "timeout while waiting for trigger to complete"? Or it gives different reasons? Can you share a few different errors ? ---
**Author:** ![adrisr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adrisr/32/25423_2.png) [@adrisr](https://discuss.elastic.co/u/adrisr)\ **Post date:** [March 2, 2022, 5:54pm UTC](https://discuss.elastic.co/t/cant-seem-to-use-socket-dataset-on-rocky-linux/298102/6 "2022-03-02T17:54:16Z")
By looking at the code, I see a reason why the trigger for both `guess_sockaddr_in` and `guess_udp_sendmsg` can timeout. Those two probes, and maybe others, require the loopback interface to be assigned the network 127.0.0.1/8. They will try to communicate using addresses like 127.123.123.123 (actually a random address). Maybe a change in your system is preventing such communication by having the loopback associated to 127.0.0.1/32 or 127.0.0.1/24. Can you check the output of any of these commands: - `ip a ls dev lo` should return: > 1: lo: \ mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 > link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 > inet **127.0.0.1/8** scope host lo > valid\_lft forever preferred\_lft forever > inet6 ::1/128 scope host > valid\_lft forever preferred\_lft forever - `ip route ls table local dev lo` > broadcast 127.0.0.0 proto kernel scope link src 127.0.0.1 > local **127.0.0.0/8** proto kernel scope host src 127.0.0.1 > local 127.0.0.1 proto kernel scope host src 127.0.0.1 > broadcast 127.255.255.255 proto kernel scope link src 127.0.0.1 And also check communication with the following commands in different terminals: (server) `nc -vln -s 127.123.123.123 -p 8888` (client) `echo hello | nc -v 127.123.123.123 8888` If communication if possible the server should output: > listening on [127.123.123.123] 8888 ... > connect to [127.123.123.123] from (UNKNOWN) [127.0.0.1] 55994 > hello ---
**Author:** ![jhbigler](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jhbigler/32/99133_2.png) [@jhbigler](https://discuss.elastic.co/u/jhbigler)\ **Post date:** [March 2, 2022, 8:57pm UTC](https://discuss.elastic.co/t/cant-seem-to-use-socket-dataset-on-rocky-linux/298102/7 "2022-03-02T20:57:23Z")
I've discovered that turning off firewalld fixes the issue on auditbeat. Obviously, we are not willing to simply turn off firewalld on all of these machines... I will continue to dig down on if it's a specific rule that's breaking the socket dataset (we use direct rules to severely limit outgoing communication, in addition to rich rules to limit inbound packets) or if it's simply the presence of firewalld. While I do this, are there any network requirements for the socket dataset to work? Other than, obviously, what is needed for any auditbeat outputs. ---
**Author:** ![jhbigler](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jhbigler/32/99133_2.png) [@jhbigler](https://discuss.elastic.co/u/jhbigler)\ **Post date:** [March 2, 2022, 9:00pm UTC](https://discuss.elastic.co/t/cant-seem-to-use-socket-dataset-on-rocky-linux/298102/8 "2022-03-02T21:00:08Z")
Also to answer @adrisr 's questions, all of those tests check out on the machine. Firewalld seems to be the culprit (see above) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 30, 2022, 11:00pm UTC](https://discuss.elastic.co/t/cant-seem-to-use-socket-dataset-on-rocky-linux/298102/9 "2022-03-30T23:00:44Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.