Grok Filter with optional fields fails


For IPTables, we log lines similar to this:

Apr 23 14:32:22 hostname1 kernel: [64719873.568549] IPTABLES-INPUT DROP :IN=br0-PROD OUT= MAC=ff:ff:ff:ff:ff:ff:52:54:00:b2:6c:4e:08:00 SRC= DST= LEN=78 TOS=0x00 PREC=0x00 TTL=128 ID=32436 PROTO=UDP SPT=137 DPT=137 LEN=58
Apr 23 14:37:07 hostname2 kernel: [59804061.226182] IPTABLES-INPUT ACCEPT: IN=ens3 OUT= MAC=01:00:5e:00:00:12:52:54:00:74:ff:22:08:00 SRC= DST= LEN=40 TOS=0x00 PREC=0xC0 TTL=255 ID=27579 PROTO=112

You will note that some lines have entries after the PROTO.

The grok filter is:

IPTABLES ((%{SYSLOGTIMESTAMP:nf_timestamp})\s*(%{HOSTNAME:nf_host})\s*kernel\S+\s*\[.*\]?.*IPTABLES-(?<nf_direction>%{WORD}) (?<nf_result>%{WORD})?.*IN=(%{USERNAME:nf_in_interface})?.*OUT=(%{USERNAME:nf_out_interface})?.*MAC=(%{COMMONMAC:nf_dst_mac}):(%{COMMONMAC:nf_src_mac})?.*SRC=(%{IPV4:nf_src_ip}).*DST=(%{IPV4:nf_dst_ip}).*PROTO=(%{WORD:nf_protocol}).?*SPT=(%{INT:nf_src_port}?.*DPT=%{INT:nf_dst_port}?.*))

This however fails to match the shorter lines. If I attempt to make these latter fields optional then all lines are grokked, BUT the optional fields are not taken, even for lines where these fields DO exist.

Is there a way - and I am sure there is - for these latter fields be grokked when necessary? Thanks

You could grok against both patterns, and use the longer pattern first. The documentation has an example of this.

You should anchor your pattern with ^ if the timestamp begins at the start of a line. It will be more efficient.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.