Filebeat won't collect syslogs over network

So I (for various reasons) would like to collect logs using Filebeat that are sent in from multiple locations on the local network. If possible I would like to access the actual logs being sent in, the actual contents of the packets, which to the best of my knowledge doesn't happen with RSyslog. The logs are being sent in to port 514 over udp. I edited the config file for Filebeat to accept logs on that port, but nothing is happening. The system module is enable, and I used setcap to give Java permissions to bind on privileged ports.

My filebeat.yml is as follows:

filebeat.inputs:
- type: log
  enabled: false
  paths:
    - /var/log/*.log
- type: udp
  enabled: true
  max_message_size: 10Kib
  host: "localhost:514"

[...]

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml

  reload.enabled: false

[...]

setup.template.settings:
  index.number_of_shards: 3

Selected portions of my Filebeat log:

2019-06-11T12:54:27.461-0500    INFO    crawler/crawler.go:72   Loading Inputs: 2
2019-06-11T12:54:27.462-0500    INFO    input/input.go:114      Starting input of type: udp; ID: 4126594670578313537
2019-06-11T12:54:27.463-0500    INFO    udp/input.go:103        Starting UDP input
2019-06-11T12:54:27.464-0500    INFO    [udp]   udp/server.go:66        Started listening for UDP connection    {"address": "localhost:514"}

Filebeat modules:

Enabled:
system

Disabled:
apache2
auditd
elasticsearch
haproxy
icinga
iis
iptables
kafka
kibana
logstash
mongodb
mysql
nginx
osquery
postgresql
redis
suricata
traefik

I have confirmed that there is constant traffic coming in on port 514. Filebeat is sending other local logs onto Logstash, so it's not a Logstash/Elastic/Kibana issue. I am stumped. There are no Filebeat logs being sent onto Logstash about any traffic. Please help.

EDIT:
I disabled the system module for Filebeat and enabled debug level logging. The only thing that changed is that it's no longer logging anything.

2019-06-11T13:45:05.140-0500    INFO    [udp]   udp/server.go:66        Started listening for UDP connection    {"address": "localhost:514"}
2019-06-11T13:45:15.140-0500    DEBUG   [input] input/input.go:152      Run input
2019-06-11T13:45:25.140-0500    DEBUG   [input] input/input.go:152      Run input
2019-06-11T13:45:35.140-0500    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":0,"time":{"ms":9}},"total":{"ticks":20,"time":{"ms":31},"value"$
2019-06-11T13:45:35.141-0500    DEBUG   [input] input/input.go:152      Run input
2019-06-11T13:45:45.141-0500    DEBUG   [input] input/input.go:152      Run input
2019-06-11T13:45:55.141-0500    DEBUG   [input] input/input.go:152      Run input
2019-06-11T13:46:05.140-0500    INFO    [monitoring]    log/log.go:144  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":0,"time":{"ms":1}},"total":{"ticks":20,"time":{"ms":5},"value":$
2019-06-11T13:46:05.141-0500    DEBUG   [input] input/input.go:152      Run input
2019-06-11T13:46:15.141-0500    DEBUG   [input] input/input.go:152      Run input

The problem persists.

EDIT:
I have also tried clearing the registry.
Still didn't work.

Hi John,

Perhaps I'm not understanding your use case correctly, but is there some reason you're using the udp input instead of the syslog input? https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-syslog.html.

Shaunak

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.