Failing to get filebeat with suricata module to work

Elastic Stack Beats
1

I have installed filebeat (7.17.1) on an ubuntu system. Run the setup stuff and loaded the dashboards into kibana.

I have also "enabled" the suricata module, first by simply adding the appropriate stuff into /etc/filebeat/modules.s/suricata and then by using the filebeat module enable and editing the resulting file.

# Module: suricata
# Docs: https://www.elastic.co/guide/en/beats/filebeat/7.17/filebeat-module-suricata.html

- module: suricata
  # All logs
  eve:
    enabled: true
    var:
      internal_networks: ["130.216.0.0/16", "202.36.244.0/23", "202.37.88.0/24",  "172.16.0.0/12", "10.0.0.0/8" ]
      paths: ["/data/sensors/eve.json"]

filebeat.yml:

setup.ilm.check_exists: true
filebeat.inputs:
filebeat.config.modules:
  path: /etc/filebeat/modules.d/*.yml
  reload.enabled: true
  reload.period: 10m
setup.template.settings:
  index.number_of_shards: 1
setup.dashboards.enabled: false
output.elasticsearch:
  hosts: ["secesprd02.its.auckland.ac.nz"]
  protocol: "https"
  api_key: "xxxxxxxxxxxxxxxxxxxxxxQ=="
processors:

log file:

rful011@secmonprd10:~$ sudo tail   /var/log/filebeat/filebeat
2022-09-21T18:49:04.181+1200	INFO	instance/beat.go:686	Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat] Hostfs Path: [/]
2022-09-21T18:49:04.181+1200	INFO	instance/beat.go:694	Beat ID: ead672a7-9fe2-47a0-8c6e-c925fde37bdb

I also tried using username and password of the user I set up to ingest the beats log ( the api key is setup to ran_as this user.

No indexes get created in ES and there are no logs on the ES server after the ones from creating the roles for the ingest user.

Baffled!

2

I have now repeated the installation on another machine with, so far as I can tell, identical configuration. This instance writes more to the log file:

2022-09-22T08:24:24.630+1200    INFO    instance/beat.go:686    Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat] Hostfs Path: [/]
2022-09-22T08:24:24.631+1200    INFO    instance/beat.go:694    Beat ID: c84062c1-3afa-4a6a-9676-a0e19a205ed7
2022-09-22T08:24:24.631+1200    INFO    [seccomp]       seccomp/seccomp.go:124  Syscall filter successfully installed
2022-09-22T08:24:24.631+1200    INFO    [beat]  instance/beat.go:1040   Beat info       {"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat"
, "uuid": "c84062c1-3afa-4a6a-9676-a0e19a205ed7"}}}
2022-09-22T08:24:24.632+1200    INFO    [beat]  instance/beat.go:1049   Build info      {"system_info": {"build": {"commit": "1d05ba86138cfc9a5ae5c0acc64a57b8d81678ff", "libbeat": "7.17.1", "time": "2022-02-23T23:38:04.000Z", "version": "7.17.1"}}}
2022-09-22T08:24:24.632+1200    INFO    [beat]  instance/beat.go:1052   Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":40,"version":"go1.17.6"}}}
2022-09-22T08:24:24.633+1200    INFO    [beat]  instance/beat.go:1056   Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2022-09-18T03:05:20+12:00","containerized":false,"name":"secmonprd14","ip":["127.0.0.1/8","fe80::10/128","f
e80::e643:4bff:fe25:1cee/64","130.216.2.68/24","fe80::e643:4bff:fe25:1cf0/64"],"kernel_version":"4.15.0-192-generic","mac":["e4:43:4b:25:1c:ee","e4:43:4b:25:1c:f0","e4:43:4b:25:1d:0e","e4:43:4b:25:1d:0f"],"os":{"type":"linux","family":"debian","platform":"ub
untu","name":"Ubuntu","version":"18.04.2 LTS (Bionic Beaver)","major":18,"minor":4,"patch":2,"codename":"bionic"},"timezone":"NZST","timezone_offset_sec":43200,"id":"33f60b3e40bf4cb4a40cbe8aeb5492da"}}}
2022-09-22T08:24:24.634+1200    INFO    [beat]  instance/beat.go:1085   Process info    {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap
","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","a
udit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_servic
e","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap
","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_r
aw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog"
,"wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/home/rful011", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 34461, "ppid": 34460, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2022-09-22T08:24
:23.480+1200"}}}
2022-09-22T08:24:24.634+1200    INFO    instance/beat.go:328    Setup Beat: filebeat; Version: 7.17.1
2022-09-22T08:24:24.634+1200    INFO    [index-management]      idxmgmt/std.go:184      Set output.elasticsearch.index to 'filebeat-7.17.1' as ILM is enabled.
2022-09-22T08:24:24.635+1200    INFO    [esclientleg]   eslegclient/connection.go:105   elasticsearch url: https://secesprd02.its.auckland.ac.nz:9200
2022-09-22T08:24:24.635+1200    INFO    [publisher]     pipeline/module.go:113  Beat name: secmonprd14
2022-09-22T08:24:24.636+1200    INFO    [monitoring]    log/log.go:142  Starting metrics logging every 30s
2022-09-22T08:24:24.636+1200    INFO    instance/beat.go:492    filebeat start running.
2022-09-22T08:24:24.637+1200    INFO    memlog/store.go:119     Loading data file of '/var/lib/filebeat/registry/filebeat' succeeded. Active transaction id=0
2022-09-22T08:24:24.637+1200    INFO    memlog/store.go:124     Finished loading transaction log file for '/var/lib/filebeat/registry/filebeat'. Active transaction id=0
2022-09-22T08:24:24.637+1200    INFO    [registrar]     registrar/registrar.go:109      States Loaded from registrar: 0
2022-09-22T08:24:24.637+1200    INFO    [crawler]       beater/crawler.go:71    Loading Inputs: 0
2022-09-22T08:24:24.637+1200    INFO    [crawler]       beater/crawler.go:108   Loading and starting Inputs completed. Enabled inputs: 0
2022-09-22T08:24:24.637+1200    INFO    cfgfile/reload.go:164   Config reloader started
2022-09-22T08:24:54.650+1200    INFO    [monitoring]    log/log.go:184  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000}},"id":"user.slice"},"cpuacct":{"id":"user.slice","total":{"ns":
90769381418303}},"memory":{"id":"user.slice","mem":{"limit":{"bytes":9223372036854771712},"usage":{"bytes":57011724288}}}},"cpu":{"system":{"ticks":110,"time":{"ms":116}},"total":{"ticks":450,"time":{"ms":461},"value":0},"user":{"ticks":340,"time":{"ms":345}
}},"handles":{"limit":{"hard":1048576,"soft":1024},"open":11},"info":{"ephemeral_id":"4061ce81-2009-4100-8446-2d3682562993","uptime":{"ms":30140},"version":"7.17.1"},"memstats":{"gc_next":19230704,"memory_alloc":12052112,"memory_sys":40977416,"memory_total":
57228168,"rss":123850752},"runtime":{"goroutines":61}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0},"type":"elasticsearch"},"pipeline":{"clients":0,"events":{"active":0},"q
ueue":{"max_events":4096}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":40},"load":{"1":6.28,"15":6.09,"5":6.2,"norm":{"1":0.157,"15":0.1523,"5":0.155}}}}}}

and I can see traffic to the ES server but I can't see any filebeat indexes in kibana.
Again there is nothing in the logs on the ES server. Since we are using https I can see what is getting sent to ES.

Do I have to have something in the input section of the config?

3

About the input section -- Docs say this is not necessary if you are using modules. I suspect that the problem is that the module file is not getting loaded -- there is no mention of it in the log file.

4

I inserted some random garbage in /etc/filebeat/modules.d/suricata.yml the should have triggered errors but filebeats started happily so it looks as if the problem that it isn't loading the module file.

I also ran

rful011@secmonprd14:~$ sudo filebeat modules enable suricata
Module suricata is already enabled

Which suggest that FB thinks that the module is enabled

5

Silly question did you run
filebeat setup -e
After you you enabled and configured the module?

Ok I played around a bit...

Agree:
I think there is an access problem with your modules.d directory or something...

I moved mine and I get the same behavior

Filebeat starts happily... no error message...

However if I made my suricata.yml unreadable then I got a permission error...

invalid config: open /Users/sbrown/workspace/elastic-install/8.3.3/filebeat-8.3.3-darwin-x86_64/modules.d/suricata.yml: permission denied

How are you starting filebeat?

What happens if you just try to start it from the command line

6

stephenbStephen BrownElastic Team Member

Silly question did you run
filebeat setup -e

After you you enabled and configured the module?
++++++++++++++++++++++++++++++++++++++++++++++++
I am starting it form the command line

and not a silly question! I know I ran the set up at one stage but running it just now returned an error "API key: invalid ApiKey value ???

I will try with creds....

Ah! now I get sensible errors about the privs of the roles. I ran setup repeatedly adding roles to the beat_setup role role until it completed. I now have the ingestion pipeline set up and the index created.

Do all the machines I install filebeat on need to contact kibana for set up. I would rather fiddle with the firewall setting for each sensor.

Anyway I now have setup completing and have gone back to running filebeat:

2022-09-22T15:19:05.805+1200    INFO    instance/beat.go:686    Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat] Hostfs Path: [/]
2022-09-22T15:19:05.805+1200    INFO    instance/beat.go:694    Beat ID: c84062c1-3afa-4a6a-9676-a0e19a205ed7
2022-09-22T15:19:05.807+1200    INFO    [seccomp]       seccomp/seccomp.go:124  Syscall filter successfully installed
2022-09-22T15:19:05.807+1200    INFO    [beat]  instance/beat.go:1040   Beat info       {"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat"
, "uuid": "c84062c1-3afa-4a6a-9676-a0e19a205ed7"}}}
2022-09-22T15:19:05.807+1200    INFO    [beat]  instance/beat.go:1049   Build info      {"system_info": {"build": {"commit": "1d05ba86138cfc9a5ae5c0acc64a57b8d81678ff", "libbeat": "7.17.1", "time": "2022-02-23T23:38:04.000Z", "version": "7.17.1"}}}
2022-09-22T15:19:05.807+1200    INFO    [beat]  instance/beat.go:1052   Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":40,"version":"go1.17.6"}}}
2022-09-22T15:19:05.809+1200    INFO    [beat]  instance/beat.go:1056   Host info       {"system_info": {"host": {"architecture":"x86_64","boot_time":"2022-09-18T03:05:20+12:00","containerized":false,"name":"secmonprd14","ip":["127.0.0.1/8","fe80::10/128","f
e80::e643:4bff:fe25:1cee/64","130.216.2.68/24","fe80::e643:4bff:fe25:1cf0/64"],"kernel_version":"4.15.0-192-generic",
untu","name":"Ubuntu","version":"18.04.2 LTS (Bionic Beaver)","major":18,"minor":4,"patch":2,"codename":"bionic"},"timezone":"NZST","timezone_offset_sec":43200,"id":"33f60b3e40bf4cb4a40cbe8aeb5492da"}}}
2022-09-22T15:19:05.809+1200    INFO    [beat]  instance/beat.go:1085   Process info    {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid",
","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","a
udit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_servic
e","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap
","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_r
aw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog"
,"wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/home/rful011", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 11129, "ppid": 11128, "seccomp": {"mode":"filter","no_new_privs":true}, "start_time": "2022-09-22T15:19
:04.680+1200"}}}
2022-09-22T15:19:05.809+1200    INFO    instance/beat.go:328    Setup Beat: filebeat; Version: 7.17.1
2022-09-22T15:19:05.809+1200    INFO    [index-management]      idxmgmt/std.go:184      Set output.elasticsearch.index to 'filebeat-7.17.1' as ILM is enabled.
2022-09-22T15:19:05.810+1200    INFO    [esclientleg]   eslegclient/connection.go:105   elasticsearch url: https://secesprd02.its.auckland.ac.nz:9200
2022-09-22T15:19:05.810+1200    INFO    [publisher]     pipeline/module.go:113  Beat name: secmonprd14
2022-09-22T15:19:05.811+1200    INFO    [monitoring]    log/log.go:142  Starting metrics logging every 30s
2022-09-22T15:19:05.811+1200    INFO    instance/beat.go:492    filebeat start running.
2022-09-22T15:19:05.812+1200    INFO    memlog/store.go:119     Loading data file of '/var/lib/filebeat/registry/filebeat' succeeded. Active transaction id=0
2022-09-22T15:19:05.812+1200    INFO    memlog/store.go:124     Finished loading transaction log file for '/var/lib/filebeat/registry/filebeat'. Active transaction id=4
2022-09-22T15:19:05.812+1200    INFO    [registrar]     registrar/registrar.go:109      States Loaded from registrar: 1
2022-09-22T15:19:05.812+1200    INFO    [crawler]       beater/crawler.go:71    Loading Inputs: 0
2022-09-22T15:19:05.812+1200    INFO    [crawler]       beater/crawler.go:108   Loading and starting Inputs completed. Enabled inputs: 0
2022-09-22T15:19:05.813+1200    INFO    cfgfile/reload.go:164   Config reloader started
2022-09-22T15:19:35.821+1200    INFO    [monitoring]    log/log.go:184  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cgroup":{"cpu":{"cfs":{"period":{"us":100000}},"id":"user.slice"},"cpuacct":{"id":"user.slice","total":{"ns":
102041692329015}},"memory":{"id":"user.slice","mem":{"limit":{"bytes":9223372036854771712},"usage":{"bytes":1335996416}}}},"cpu":{"system":{"ticks":160,"time":{"ms":169}},"total":{"ticks":380,"time":{"ms":391},"value":380},"user":{"ticks":220,"time":{"ms":22
2}}},"handles":{"limit":{"hard":1048576,"soft":1024},"open":11},"info":{"ephemeral_id":"0ca8d2b4-c16a-43d4-8f03-ea42ef6a9613","uptime":{"ms":30111},"version":"7.17.1"},"memstats":{"gc_next":19253904,"memory_alloc":11836600,"memory_sys":41763848,"memory_total
":57014512,"rss":124764160},"runtime":{"goroutines":61}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":0},"type":"elasticsearch"},"pipeline":{"clients":0,"events":{"active":0},
"queue":{"max_events":4096}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":40},"load":{"1":9.35,"15":10.23,"5":10.02,"norm":{"1":0.2338,"15":0.2558,"5":0.2505}}}}}}

and the monitoring entries continue but no actual logs are shipped

7

No you only need to run setup once... that is when Kibana is contacted.
After that filebeat only outputs to elasticsearch.

So run setup once per version / version upgrade (needs to contact Kibana)
Then run filebeat on 1000 host, which only outputs to elasticsearch.

Yes it seems it is not loading the module you should see something like and see your path in it.
Don't for get to clean up the registry to reload the data...

022-09-21T21:25:19.002-0700    WARN    [cfgwarn]       log/input.go:89 DEPRECATED: Log input. Use Filestream input instead.
2022-09-21T21:25:19.002-0700    INFO    [input] log/input.go:171        Configured paths: [/usr/local/var/log/suricata/eve.json]        {"input_id": "0cc9ec4e-2507-4695-8352-dc2409f68f22"}

Try this

# filebeat.config.modules:
#   # Glob pattern for configuration loading
#   path: ${path.config}/modules.d/*.yml

#   # Set to true to enable config reloading
#   reload.enabled: false

filebeat.modules:
  - module: suricata
    # All logs
    eve:
      enabled: true
      var:
        internal_networks: ["130.216.0.0/16", "202.36.244.0/23", "202.37.88.0/24",  "172.16.0.0/12", "10.0.0.0/8" ]
        paths: ["/data/sensors/eve.json"]
8

Thanks Stephen, Our my colleague who is preparing our quarterly PCI report will be very relieved. : )

That's what I thought -- nothing else made sense.

on the config front I did try putting the module config in the. main file but did not get it quite right! I now have data going in to ES!

Should I report this as an issue? I.e that the modules.d mechanism appears to be non functional at least on my ubuntu systems.

There are a few more niggles but if they are still issues tomorrow I will raise a new thread

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.