Grok Filter with optional fields fails

[harrytewkesbury (harry.tewkesbury@worldfirst.com)]

Hi,

For IPTables, we log lines similar to this:

Apr 23 14:32:22 hostname1 kernel: [64719873.568549] IPTABLES-INPUT DROP :IN=br0-PROD OUT= MAC=ff:ff:ff:ff:ff:ff:52:54:00:b2:6c:4e:08:00 SRC=10.3.9.21 DST=10.3.9.255 LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=32436 PROTO=UDP SPT=137 DPT=137 LEN=58
Apr 23 14:37:07 hostname2 kernel: [59804061.226182] IPTABLES-INPUT ACCEPT: IN=ens3 OUT= MAC=01:00:5e:00:00:12:52:54:00:74:ff:22:08:00 SRC=10.3.5.2 DST=224.0.0.18 LEN=40 TOS=0x00 PREC=0xC0 TTL=255 ID=27579 PROTO=112

You will note that some lines have entries after the PROTO.

The grok filter is:

IPTABLES ((%{SYSLOGTIMESTAMP:nf_timestamp})\s*(%{HOSTNAME:nf_host})\s*kernel\S+\s*\[.*\]?.*IPTABLES-(?<nf_direction>%{WORD}) (?<nf_result>%{WORD})?.*IN=(%{USERNAME:nf_in_interface})?.*OUT=(%{USERNAME:nf_out_interface})?.*MAC=(%{COMMONMAC:nf_dst_mac}):(%{COMMONMAC:nf_src_mac})?.*SRC=(%{IPV4:nf_src_ip}).*DST=(%{IPV4:nf_dst_ip}).*PROTO=(%{WORD:nf_protocol}).?*SPT=(%{INT:nf_src_port}?.*DPT=%{INT:nf_dst_port}?.*))

This however fails to match the shorter lines. If I attempt to make these latter fields optional then all lines are grokked, BUT the optional fields are not taken, even for lines where these fields DO exist.

Is there a way - and I am sure there is - for these latter fields be grokked when necessary? Thanks

[Badger]

You could grok against both patterns, and use the longer pattern first. The documentation has an example of this.

You should anchor your pattern with ^ if the timestamp begins at the start of a line. It will be more efficient.