No logs in discovery

Hello,
I have installed the following on three VMs:

  • Elasticsearch: 192.168.101.9

  • Kibana: 192.168.101.7

  • OPNsense: 192.168.101.8

Next, I started the Fleet Server and installed it on Kibana (192.168.101.7). I then ran two agents (with configuration details visible in the screenshot) so they could collect logs from OPNsense and make them visible in the Discovery view in Kibana.

However, even though I can see the logs flowing into the Kibana machine (192.168.101.7), neither of the agents seems to be picking them up. I suspect this because the two following commands return no results:

  • sudo journalctl -u elastic-agent -n 100 --no-pager | grep 9002

  • sudo journalctl -u elastic-agent -n 100 --no-pager | grep 9003

I am attaching several screenshots and configuration commands below.

sudo ss -lunp | grep -E "9002|9003"

UNCONN 0 0 [::]:9002 [::]:* users:(("agentbeat",pid=27722,fd=8))

UNCONN 0 0 [::]:9003 [::]:* users:(("agentbeat",pid=27722,fd=7))

sudo tcpdump -i any udp port 9002 -n

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes

11:22:59.599083 IP 192.168.101.8.upstriggervsw > 192.168.101.7.dynamid: UDP, length 226

11:22:59.599138 IP 192.168.101.8.upstriggervsw > 192.168.101.7.dynamid: UDP, length 220

11:22:59.599156 IP 192.168.101.8.upstriggervsw > 192.168.101.7.dynamid: UDP, length 226

11:22:59.599171 IP 192.168.101.8.upstriggervsw > 192.168.101.7.dynamid: UDP, length 226

sudo tcpdump -i any udp port 9003 -n

tcpdump: verbose output suppressed, use -v or -vv for full protocol decode

listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes

11:24:17.042978 IP 192.168.101.8.14028 > 192.168.101.7.9003: UDP, length 192

11:24:17.043032 IP 192.168.101.8.14028 > 192.168.101.7.9003: UDP, length 186

11:24:17.043050 IP 192.168.101.8.14028 > 192.168.101.7.9003: UDP, length 188

11:24:17.043067 IP 192.168.101.8.14028 > 192.168.101.7.9003: UDP, length 191

Hello @NeedHelp

Welcome to the Community!!

To start with could you please confirm if the system integration is working as expected , are we seeing the system logs/metrics for both the Fleet Server & Fleet Agent in Kibana? If this is working means ElasticAgent is able to send the logs/metrics correctly. Post that we can try to troubleshoot the issue with the Integration : OPNsense

Also if the understanding is correct below is your current setup ?

Elasticsearch: 192.168.101.9
Kibana: 192.168.101.7 & (Fleet Server Installed)
OPNsense: 192.168.101.8 & (Fleet Agent Installed + Data for OPNsense needs to be collected from this host)

OPNsense Setup
Navigate to System -> Settings -> Logging/Targets
Add a new Logging/Target (Click the plus icon)
Transport = UDP or TCP or TLS
Applications = Select a list of applications to send to remote syslog. Leave empty for all.
Levels = Nothing Selected
Facilities = Nothing Selected
Hostname = IP of Elastic agent as configured in the integration config
Port = Port of Elastic agent as configured in the integration config
Certificate = Client certificate to use (when selecting a tls transport type)
Description = Syslog to Elasticsearch
Click Save

Thanks!!