Need help with custom Filebeat modules, no input data in real tests but local dev test yes

Hi there,
i created my own filebeat module, "filebeat-modules-devguide" served as the basis.
All tests had been successful and now wanted to test them in real.
The index and the ingest pipelines are created successfully, also a UDP server is started on the corresponding filebeat port. However, no data is received from the UDP port, even though it is sent and also arrives (tested with tcpdump).

2020-06-02T17:51:13.013+0200	INFO	cfgfile/reload.go:164	Config reloader started
2020-06-02T17:51:13.013+0200	DEBUG	[cfgfile]	cfgfile/reload.go:194	Scan for new config files
2020-06-02T17:51:13.013+0200	DEBUG	[cfgfile]	cfgfile/cfgfile.go:193	Load config from file: /etc/filebeat/modules.d/sophosxg.yml
2020-06-02T17:51:13.013+0200	DEBUG	[cfgfile]	cfgfile/reload.go:213	Number of module configs found: 1
2020-06-02T17:51:13.013+0200	DEBUG	[reload]	cfgfile/list.go:62	Starting reload procedure, current runners: 0
2020-06-02T17:51:13.013+0200	DEBUG	[reload]	cfgfile/list.go:80	Start list: 1, Stop list: 0
2020-06-02T17:51:13.014+0200	DEBUG	[processors]	processors/processor.go:101	Generated new processors: add_locale=[format=offset]
2020-06-02T17:51:13.014+0200	DEBUG	[reload]	cfgfile/list.go:101	Starting runner: sophosxg (firewall)
2020-06-02T17:51:13.014+0200	INFO	eslegclient/connection.go:97	elasticsearch url:
2020-06-02T17:51:13.014+0200	WARN	[tls]	tlscommon/tls_config.go:83	SSL/TLS verifications disabled.
2020-06-02T17:51:13.014+0200	DEBUG	[esclientleg]	eslegclient/connection.go:284	ES Ping(url=
2020-06-02T17:51:13.015+0200	WARN	[tls]	tlscommon/tls_config.go:83	SSL/TLS verifications disabled.
2020-06-02T17:51:13.036+0200	DEBUG	[esclientleg]	eslegclient/connection.go:307	Ping status code: 200
2020-06-02T17:51:13.036+0200	INFO	[esclientleg]	eslegclient/connection.go:308	Attempting to connect to Elasticsearch version 7.7.0
2020-06-02T17:51:13.036+0200	DEBUG	[modules]	fileset/pipelines.go:67	Required processors: [{geoip ingest-geoip}]
2020-06-02T17:51:13.036+0200	DEBUG	[esclientleg]	eslegclient/connection.go:358	GET  <nil>
2020-06-02T17:51:13.051+0200	DEBUG	[esclientleg]	eslegclient/connection.go:358	GET  <nil>
2020-06-02T17:51:13.056+0200	DEBUG	[modules]	fileset/pipelines.go:120	Pipeline filebeat-8.0.0-sophosxg-firewall-pipeline already loaded
2020-06-02T17:51:13.056+0200	DEBUG	[esclientleg]	eslegclient/connection.go:358	GET  <nil>
2020-06-02T17:51:13.062+0200	DEBUG	[modules]	fileset/pipelines.go:120	Pipeline filebeat-8.0.0-sophosxg-firewall-firewall already loaded
2020-06-02T17:51:13.062+0200	DEBUG	[esclientleg]	eslegclient/connection.go:358	GET  <nil>
2020-06-02T17:51:13.065+0200	DEBUG	[modules]	fileset/pipelines.go:120	Pipeline filebeat-8.0.0-sophosxg-firewall-idp already loaded
2020-06-02T17:51:13.065+0200	DEBUG	[esclientleg]	eslegclient/connection.go:358	GET  <nil>
2020-06-02T17:51:13.069+0200	DEBUG	[modules]	fileset/pipelines.go:120	Pipeline filebeat-8.0.0-sophosxg-firewall-atp already loaded
2020-06-02T17:51:13.069+0200	DEBUG	[esclientleg]	eslegclient/connection.go:358	GET  <nil>
2020-06-02T17:51:13.073+0200	DEBUG	[modules]	fileset/pipelines.go:120	Pipeline filebeat-8.0.0-sophosxg-firewall-antivirus already loaded
2020-06-02T17:51:13.073+0200	DEBUG	[esclientleg]	eslegclient/connection.go:358	GET  <nil>
2020-06-02T17:51:13.075+0200	DEBUG	[modules]	fileset/pipelines.go:120	Pipeline filebeat-8.0.0-sophosxg-firewall-sandstorm already loaded
2020-06-02T17:51:13.075+0200	DEBUG	[esclientleg]	eslegclient/connection.go:358	GET  <nil>
2020-06-02T17:51:13.079+0200	DEBUG	[modules]	fileset/pipelines.go:120	Pipeline filebeat-8.0.0-sophosxg-firewall-cfilter already loaded
2020-06-02T17:51:13.079+0200	DEBUG	[esclientleg]	eslegclient/connection.go:358	GET  <nil>
2020-06-02T17:51:13.082+0200	DEBUG	[modules]	fileset/pipelines.go:120	Pipeline filebeat-8.0.0-sophosxg-firewall-event already loaded
2020-06-02T17:51:13.082+0200	INFO	cfgfile/reload.go:224	Loading of config files completed.
2020-06-02T17:51:13.082+0200	INFO	udp/input.go:103	Starting UDP input


module_version: 1.0

  - name: syslog_host
    default: localhost
  - name: tags
    default: [sophosxg-firewall]
  - name: syslog_port
    default: 9005
  - name: input
    default: udp

  - ingest/pipeline.yml
  - ingest/firewall.yml
  - ingest/idp.yml
  - ingest/atp.yml
  - ingest/antivirus.yml
  - ingest/sandstorm.yml
  - ingest/cfilter.yml
  - ingest/event.yml

input: config/firewall.yml

- name: geoip
  plugin: ingest-geoip

config input.yml:

{{ if eq .input "tcp" }}

type: tcp
host: "{{.syslog_host}}:{{.syslog_port}}"

{{ else if eq .input "udp" }}

type: udp
host: "{{.syslog_host}}:{{.syslog_port}}"

{{ else if eq .input "file" }}

type: log
{{ range $i, $path := .paths }}
  - {{$path}}
{{ end }}

exclude_files: [".gz$"]

{{ end }}

tags: {{.tags}}

  - add_locale: ~

What am i doing wrong here ?
Thanks for any help.


No idea what that could be?
Is there a way to test more closely here to narrow down the problem?


Could you please share the module configuration as well?

Hi Noemi,
you mean the content of the pipelines or something else?

you mean this ?

# Module: sophosxg
# Docs:

- module: sophosxg
    enabled: true

    # Set which input to use between tcp, udp (default) or file.
    #var.input: udp

    # The interface to listen to syslog traffic. Defaults to
    # localhost. Set to to bind to all available interfaces.

    # The port to listen for syslog traffic. Defaults to 9005.
    #var.syslog_port: 9005

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.