Need help with custom Filebeat modules, no input data in real tests but local dev test yes

Elastic Stack Beats
, ,
1

Hi there,
i created my own filebeat module, "filebeat-modules-devguide" served as the basis.
All tests had been successful and now wanted to test them in real.
The index and the ingest pipelines are created successfully, also a UDP server is started on the corresponding filebeat port. However, no data is received from the UDP port, even though it is sent and also arrives (tested with tcpdump).

2020-06-02T17:51:13.013+0200	INFO	cfgfile/reload.go:164	Config reloader started
2020-06-02T17:51:13.013+0200	DEBUG	[cfgfile]	cfgfile/reload.go:194	Scan for new config files
2020-06-02T17:51:13.013+0200	DEBUG	[cfgfile]	cfgfile/cfgfile.go:193	Load config from file: /etc/filebeat/modules.d/sophosxg.yml
2020-06-02T17:51:13.013+0200	DEBUG	[cfgfile]	cfgfile/reload.go:213	Number of module configs found: 1
2020-06-02T17:51:13.013+0200	DEBUG	[reload]	cfgfile/list.go:62	Starting reload procedure, current runners: 0
2020-06-02T17:51:13.013+0200	DEBUG	[reload]	cfgfile/list.go:80	Start list: 1, Stop list: 0
2020-06-02T17:51:13.014+0200	DEBUG	[processors]	processors/processor.go:101	Generated new processors: add_locale=[format=offset]
2020-06-02T17:51:13.014+0200	DEBUG	[reload]	cfgfile/list.go:101	Starting runner: sophosxg (firewall)
2020-06-02T17:51:13.014+0200	INFO	eslegclient/connection.go:97	elasticsearch url: https://172.16.34.221:9200
2020-06-02T17:51:13.014+0200	WARN	[tls]	tlscommon/tls_config.go:83	SSL/TLS verifications disabled.
2020-06-02T17:51:13.014+0200	DEBUG	[esclientleg]	eslegclient/connection.go:284	ES Ping(url=https://172.16.34.221:9200)
2020-06-02T17:51:13.015+0200	WARN	[tls]	tlscommon/tls_config.go:83	SSL/TLS verifications disabled.
2020-06-02T17:51:13.036+0200	DEBUG	[esclientleg]	eslegclient/connection.go:307	Ping status code: 200
2020-06-02T17:51:13.036+0200	INFO	[esclientleg]	eslegclient/connection.go:308	Attempting to connect to Elasticsearch version 7.7.0
2020-06-02T17:51:13.036+0200	DEBUG	[modules]	fileset/pipelines.go:67	Required processors: [{geoip ingest-geoip}]
2020-06-02T17:51:13.036+0200	DEBUG	[esclientleg]	eslegclient/connection.go:358	GET https://172.16.34.221:9200/_nodes/ingest  <nil>
2020-06-02T17:51:13.051+0200	DEBUG	[esclientleg]	eslegclient/connection.go:358	GET https://172.16.34.221:9200/_ingest/pipeline/filebeat-8.0.0-sophosxg-firewall-pipeline  <nil>
2020-06-02T17:51:13.056+0200	DEBUG	[modules]	fileset/pipelines.go:120	Pipeline filebeat-8.0.0-sophosxg-firewall-pipeline already loaded
2020-06-02T17:51:13.056+0200	DEBUG	[esclientleg]	eslegclient/connection.go:358	GET https://172.16.34.221:9200/_ingest/pipeline/filebeat-8.0.0-sophosxg-firewall-firewall  <nil>
2020-06-02T17:51:13.062+0200	DEBUG	[modules]	fileset/pipelines.go:120	Pipeline filebeat-8.0.0-sophosxg-firewall-firewall already loaded
2020-06-02T17:51:13.062+0200	DEBUG	[esclientleg]	eslegclient/connection.go:358	GET https://172.16.34.221:9200/_ingest/pipeline/filebeat-8.0.0-sophosxg-firewall-idp  <nil>
2020-06-02T17:51:13.065+0200	DEBUG	[modules]	fileset/pipelines.go:120	Pipeline filebeat-8.0.0-sophosxg-firewall-idp already loaded
2020-06-02T17:51:13.065+0200	DEBUG	[esclientleg]	eslegclient/connection.go:358	GET https://172.16.34.221:9200/_ingest/pipeline/filebeat-8.0.0-sophosxg-firewall-atp  <nil>
2020-06-02T17:51:13.069+0200	DEBUG	[modules]	fileset/pipelines.go:120	Pipeline filebeat-8.0.0-sophosxg-firewall-atp already loaded
2020-06-02T17:51:13.069+0200	DEBUG	[esclientleg]	eslegclient/connection.go:358	GET https://172.16.34.221:9200/_ingest/pipeline/filebeat-8.0.0-sophosxg-firewall-antivirus  <nil>
2020-06-02T17:51:13.073+0200	DEBUG	[modules]	fileset/pipelines.go:120	Pipeline filebeat-8.0.0-sophosxg-firewall-antivirus already loaded
2020-06-02T17:51:13.073+0200	DEBUG	[esclientleg]	eslegclient/connection.go:358	GET https://172.16.34.221:9200/_ingest/pipeline/filebeat-8.0.0-sophosxg-firewall-sandstorm  <nil>
2020-06-02T17:51:13.075+0200	DEBUG	[modules]	fileset/pipelines.go:120	Pipeline filebeat-8.0.0-sophosxg-firewall-sandstorm already loaded
2020-06-02T17:51:13.075+0200	DEBUG	[esclientleg]	eslegclient/connection.go:358	GET https://172.16.34.221:9200/_ingest/pipeline/filebeat-8.0.0-sophosxg-firewall-cfilter  <nil>
2020-06-02T17:51:13.079+0200	DEBUG	[modules]	fileset/pipelines.go:120	Pipeline filebeat-8.0.0-sophosxg-firewall-cfilter already loaded
2020-06-02T17:51:13.079+0200	DEBUG	[esclientleg]	eslegclient/connection.go:358	GET https://172.16.34.221:9200/_ingest/pipeline/filebeat-8.0.0-sophosxg-firewall-event  <nil>
2020-06-02T17:51:13.082+0200	DEBUG	[modules]	fileset/pipelines.go:120	Pipeline filebeat-8.0.0-sophosxg-firewall-event already loaded
2020-06-02T17:51:13.082+0200	INFO	cfgfile/reload.go:224	Loading of config files completed.
2020-06-02T17:51:13.082+0200	INFO	udp/input.go:103	Starting UDP input

manifest:

module_version: 1.0

var:
  - name: syslog_host
    default: localhost
  - name: tags
    default: [sophosxg-firewall]
  - name: syslog_port
    default: 9005
  - name: input
    default: udp

ingest_pipeline:
  - ingest/pipeline.yml
  - ingest/firewall.yml
  - ingest/idp.yml
  - ingest/atp.yml
  - ingest/antivirus.yml
  - ingest/sandstorm.yml
  - ingest/cfilter.yml
  - ingest/event.yml

input: config/firewall.yml

requires.processors:
- name: geoip
  plugin: ingest-geoip

config input.yml:

{{ if eq .input "tcp" }}

type: tcp
host: "{{.syslog_host}}:{{.syslog_port}}"

{{ else if eq .input "udp" }}

type: udp
host: "{{.syslog_host}}:{{.syslog_port}}"

{{ else if eq .input "file" }}

type: log
paths:
{{ range $i, $path := .paths }}
  - {{$path}}
{{ end }}

exclude_files: [".gz$"]

{{ end }}

tags: {{.tags}}

processors:
  - add_locale: ~

What am i doing wrong here ?
Thanks for any help.

StefanS

2

No idea what that could be?
Is there a way to test more closely here to narrow down the problem?

thx
StefanS

3

Could you please share the module configuration as well?

4

Hi Noemi,
you mean the content of the pipelines or something else?

5

Noemi,
you mean this ?

# Module: sophosxg
# Docs: https://www.elastic.co/guide/en/beats/filebeat/master/filebeat-module-sophosxg.html

- module: sophosxg
  firewall:
    enabled: true

    # Set which input to use between tcp, udp (default) or file.
    #var.input: udp

    # The interface to listen to syslog traffic. Defaults to
    # localhost. Set to 0.0.0.0 to bind to all available interfaces.
    var.syslog_host: 0.0.0.0

    # The port to listen for syslog traffic. Defaults to 9005.
    #var.syslog_port: 9005
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.