Hi,
I try to filter messages in the filebeat module section to parse a single logstream into system and iptables parsed logs.
Syslog is received from our linux based (openwrt to be specific) devices over the network and stored to file locally with rsyslog. Filebeat picks up the local logs and should preparse them through system and iptables modules.
An example per device logfile looks like this.
May 21 14:27:11 tom-tel netifd: wan (1075): udhcpc: sending renew to 192.168.0.1
May 21 14:27:11 tom-tel netifd: wan (1075): udhcpc: lease of 192.168.0.136 obtained, lease time 3600
May 21 14:39:00 tom-tel crond[931]: USER root pid 29994 cmd /usr/sbin/nw-signal
May 21 14:39:21 tom-tel kernel: [756813.361341] REJECT wan in: IN=eth0.2 OUT= MAC=ff:ff:ff:ff:ff:ff:d4:81:d7:7a:0b:06:08:00:45:00:01:68 SRC=0.0.0.0 DST=255.255.255.255 LEN=360 TOS=0x00 PREC=0x00 TTL=64 ID=25352 PROTO=UDP SPT=68 DPT=67 LEN=340
My config for the system module looks like this. Unfortunately no success so far.
Logline:
Jun 25 08:35:33 kandern-wifi2wifi-a100m kernel: [312555.373278] REJECT wan in: IN=wlan0 OUT= MAC=94:83:c4:02:7c:2c:38:10:d5:4a:a2:f3:08:00 SRC=192.168.178.1 DST=192.168.178.68 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=26962 DF PROTO=TCP SPT=472
96 DPT=80 WINDOW=14600 RES=0x00 SYN URGP=0
Config:
- module: system
# Syslog
syslog:
enabled: true
var.paths: ["/var/log/filebeat/narrowin/*"]
input.processors:
- drop_event.when.contains.message: "REJECT"
filebeat debug log:
2020-06-25T08:35:42.106+0200 DEBUG [publisher] pipeline/client.go:220 Pipeline client receives callback 'onFilteredOut' for event: {Timestamp:2020-06-25 08:35:37.105491598 +0200 CEST m=+41.850465821 Meta:{"pipeline":"filebeat-7.8.0-system-syslog-pipeline"} Fields:{"ecs":{"version":"1.5.0"},"event":{"dataset":"system.syslog","module":"system","timezone":"+02:00"},"fileset":{"name":"syslog"},"input":{"type":"log"},"log":{"file":{"path":"/var/log/filebeat/narrowin/kandern-wifi2wifi-a100m.log"},"offset":11690382},"message":"Jun 25 08:35:33 kandern-wifi2wifi-a100m kernel: [312555.373278] REJECT wan in: IN=wlan0 OUT= MAC=94:83:c4:02:7c:2c:38:10:d5:4a:a2:f3:08:00 SRC=192.168.178.1 DST=192.168.178.68 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=26962 DF PROTO=TCP SPT=47296 DPT=80 WINDOW=14600 RES=0x00 SYN URGP=0","service":{"type":"system"}} Private:{Id: Finished:false Fileinfo:0xc00079add0 Source:/var/log/filebeat/narrowin/kandern-wifi2wifi-a100m.log Offset:11690663 Timestamp:2020-06-25 08:34:58.981880341 +0200 CEST m=+3.726854576 TTL:-1ns Type:log Meta:map[] FileStateOS:98042187-64768} TimeSeries:false}
2020-06-25T08:35:42.106+0200 DEBUG [acker] beater/acker.go:64 stateful ack {"count": 1}
2020-06-25T08:35:42.106+0200 DEBUG [registrar] registrar/registrar.go:356 Processing 1 events
2020-06-25T08:35:42.106+0200 DEBUG [registrar] registrar/registrar.go:326 Registrar state updates processed. Count: 1
2020-06-25T08:35:42.106+0200 DEBUG [registrar] registrar/registrar.go:411 Write registry file: /var/lib/filebeat/registry/filebeat/data.json (10)
2020-06-25T08:35:42.172+0200 DEBUG [reader_multiline] multiline/multiline.go:177 Multiline event flushed because timeout reached.
2020-06-25T08:35:42.172+0200 DEBUG [add_docker_metadata] add_docker_metadata/add_docker_metadata.go:206 Container not found: cid=kandern-wifi2wifi-a100m.log
2020-06-25T08:35:42.172+0200 DEBUG [processors] processing/processors.go:187 Publish event: {
"@timestamp": "2020-06-25T06:35:37.171Z",
"@metadata": {
In kibana I still see the REJECT line appear under the system logs. Any idea why this is not dropped?
Thx 1000x
Mischa