I've set up a OPNsense which is successfully communicating with ELK (running in docker, GitHub - peasead/elastic-container: Stand up a simple Elastic container with Kibana, Fleet, and the Detection Engine) as both filterlogs & dhcp logs are being ingested in ELK and present in the discover tab, however both suricata logs and unbound DNS logs are not present.
pfsense logs which are present
According to OPNsense no logs are being dropped and all logs should be written to the target. (yes i'm using syslog)
dst.network d_c51a4e53b30e4d70b2c74175588f3db7#0 tcp,redacted:9001 o written 522 unbound
dst.network d_c51a4e53b30e4d70b2c74175588f3db7#0 tcp,redacted:9001 o truncated_count 0 unbound
dst.network d_c51a4e53b30e4d70b2c74175588f3db7#0 tcp,redacted:9001 o truncated_bytes 0 unbound
dst.network d_c51a4e53b30e4d70b2c74175588f3db7#0 tcp,redacted:9001 o queued 0 unbound
destination d_c51a4e53b30e4d70b2c74175588f3db7 o processed 522 unbound
dst.network d_c51a4e53b30e4d70b2c74175588f3db7#0 tcp,redacted:9001 o processed 522 unbound
dst.network d_c51a4e53b30e4d70b2c74175588f3db7#0 tcp,redacted:9001 o msg_size_max 205 unbound
dst.network d_c51a4e53b30e4d70b2c74175588f3db7#0 tcp,redacted:9001 o msg_size_avg 155 unbound
dst.network d_c51a4e53b30e4d70b2c74175588f3db7#0 tcp,redacted:9001 o memory_usage 0 unbound
dst.network d_c51a4e53b30e4d70b2c74175588f3db7#0 tcp,redacted:9001 o eps_since_start 0 unbound
dst.network d_c51a4e53b30e4d70b2c74175588f3db7#0 tcp,redacted:9001 o eps_last_24h 0 unbound
dst.network d_c51a4e53b30e4d70b2c74175588f3db7#0 tcp,redacted:9001 o eps_last_1h 0 unbound
dst.network d_c51a4e53b30e4d70b2c74175588f3db7#0 tcp,redacted:9001 o dropped 0 unbound
I've tried multiple different configurations (tcp/udp, everything selects, nothing selected, only a few services selected, ...) in OPNSense to send the logs to test what may be the problem without much success. If I select everything only the filterlog & dhcp logs are present and all logs should be written (according to OPNSense). Testing only Unbound, OPNsense still reports logs being written but no logs are present in ELK.
During the debugging steps I've looked at many places, tutorials to set up the logging and also the pfelk documentation GitHub - pfelk/pfelk: pfSense/OPNsense + Elastic Stack (i have imported the configuration settings for unbound and suricata as mentioned in this repo). Everything I've set up seems like it should work but the unbound logs are not present.
For suricata itself I read that this is not supported by the pfsenses integration https://docs.elastic.co/en/integrations/pfsense which I am using so that could explain why it's not present (although OPNsense does not report the logs dropped) but according to the integration unbound logs should be collected.
From what I've been able to deduce I'm guessing the issue is somewhere in ELK that the logs are either not parsed or just out right dropped
- Is there a specific place where I can look in certain debug/log files to see what might be causing the issue or give me more information on what is going on?
- Any suggestions on what else I could try to solve this issue?
Kind regards 