Is my data just not conforming to the pipeline? Filebeat not processing iptables

Hopefully the group can help me out here... I've got my network infrastructure (coming out of a unifi controller) sending syslog to a filebeat over UDP/514. The messages are iptables logs. I'm not seeing any of the messages get processed into elasticsearch. I DID confirm that messages are coming into the filebeat host on 514 with a tcpdump. I also set up syslog-ng and created a test file for filebeat and tried to process it using "read from file" instead of "read from syslog". And even that fails...

Here's what my filebeat.yml looks like:

      logging.to_files: false
         path: ${path.config}/modules.d/*.yml
         reload.enabled: false

    logging.level: debug

       hosts: '${ELASTICSEARCH_HOSTS}'
       username: '${ELASTICSEARCH_USERNAME}'
       password: '${ELASTICSEARCH_PASSWORD}'
       host: "kib01:5601"

And here's my iptables.yml

    # Module: iptables
    # Docs:

    - module: iptables
         enabled: true
         var.convert_timezone: true
         var.syslog_host: localhost
         var.syslog_port: 514
         # Set which input to use between syslog (default) or file.
         var.input: "syslog"

And here's a sample of the log data that is being sent:

    Jan 26 09:18:47 30c04bee8c30,udm- kernel: [322287.006104] IN=br30 OUT=eth4 MAC=f6:92:bf:7f:a0:e8:40:b4:cd:28:95:40:08:00 SRC= DST= LEN=136 TOS=0x00 PREC=0x00 TTL=63 ID=39401 DF PROTO=TCP SPT=37398 DPT=80 WINDOW=274 RES=0x00 ACK PSH URGP=0 
    Jan 26 09:18:47 30c04bee8c30,udm- kernel: [322287.020222] IN=br10 OUT=eth4 MAC=f6:92:bf:7f:a0:e8:5c:3a:45:9a:c1:8d:08:00 SRC= DST= LEN=62 TOS=0x00 PREC=0x00 TTL=127 ID=38007 DF PROTO=UDP SPT=51931 DPT=443 LEN=42

Any help appreciated!

hi @rfischman, can you check the Filebeat logs, maybe they contain more information on what the issue is. For a better look can enable debug level for logging so you can see if any events are being processed?

So I'm not sure what happened but when I restarted it with with debug mode enabled, everything started working... I then switched it back to run without debugging and it continued to function.

Very odd behavior, but I'm not going to argue with it!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.