PFsense integration cannot parse OPNSense ISC DHCP logs properly

Elastic Stack Elastic Agent
1

Hello,

The PFSense integration cannot parse OPNSense DHCPv4 logs properly.

OS: Debian 11.10 (x86_64)
PFSense integration: 1.19.1
ELK: 8.14.3
Elastic Agent: 8.14.3
OPNSense: 24.7_9
ISC DHCPv4: 4.4.3P1_1

For example, the following messages are not properly parsed:

DHCPREQUEST for 1.2.3.4 from 01:02:03:04:05:06 via igb0_vlan10
DHCPACK on 1.2.3.4 to 01:02:03:04:05:06 via igb0_vlan10
DHCPDISCOVER from 01:02:03:04:05:06 via igb0_vlan10
DHCPREQUEST for 1.2.3.4 (5.6.7.8) from 01:02:03:04:05:06 (CLIENT01) via igb0_vlan10
DHCPOFFER on 1.2.3.4 to 01:02:03:04:05:06 (CLIENT01) via igb0_vlan10

I think that the grok patterns for the DHCP pipeline might not be right:
/packages/pfsense/data_stream/log/elasticsearch/ingest_pipeline/dhcp.yml

I would probably write it like the following

proposed modifications 
  - grok:
      field: message
      patterns:
        - '%{DATA:_tmp.action}/%{INTERFACE:observer.ingress.interface.name}/%{MAC_ADDRESS:server.mac}/%{NOTSPACE:pfsense.dhcp.subnet}'
        - '%{DATA:_tmp.action} %{IPV6:client.address}(/%{NUMBER})? on %{INTERFACE:observer.ingress.interface.name}'
        - '%{DATA:_tmp.action} (from|to) %{IPV6:client.address} port %{POSINT:client.port:long}(, transaction ID %{NOTSPACE:pfsense.dhcp.transaction_id})?'
        - '%{DATA:_tmp.action} for: %{IPV6:client.address}(, age %{POSINT:pfsense.dhcp.age:long} secs)?%{GREEDYDATA}'
        - '%{DATA:_tmp.action}: address %{IPV6:client.address} to client with duid %{DUID:pfsense.dhcp.duid} iaid = -%{NOTSPACE:pfsense.dhcp.iaid} valid for %{POSINT:pfsense.dhcp.lease_time:long} seconds'
        - '%{WORD:_tmp.action} %{MIDDLE} via %{INTERFACE:observer.ingress.interface.name}'
        - '%{DATA:_tmp.action} %{IPV6:client.address}'
        - '%{GREEDYDATA}'
      pattern_definitions:
        INTERFACE: '[a-z0-9_\.]+'
        MAC_ADDRESS: '([0-9A-Fa-f]{2}[:-]){5}([0-9A-Fa-f]{2})'
        FROM: 'from %{MAC_ADDRESS:client.mac}'
        ON: 'on %{IP:client.address} to %{MAC_ADDRESS:client.mac}(?: \(%{HOSTNAME:pfsense.dhcp.hostname}\))?'
        FOR: 'for %{IP:client.address}(?: \(%{IP:server.address}\))? from %{MAC_ADDRESS:client.mac}(?: \(%{HOSTNAME:pfsense.dhcp.hostname}\))?'
        MIDDLE: '(?:%{FROM}|%{ON}|%{FOR})'
        DUID: '(?i)[0-9a-f]{2}(:[0-9a-f]{2})+'
2

Thank you @pcollardez for the discussion. Looking at the pfsense docs page the pfsense integration was graciously donated to use from the community. We have a level of support of Community and we can reach out to the original authors for modification guidance. We would absolutely welcome contributions into the code base within the pfsense package.