Filebeat setup reports missing module/suricata

Elastic Stack Beats
1

Hey folks! :wave:

It's my first post here, so please excuse any mistakes I might make.

I've started to set up Filebeat, Elastic and Kibana to connect Suricata to Grafana two days ago. Yesterday night at 3am I finally thought that everything is working buuuttt ... It didn't :melting_face:

My Suricata Dashboard at https://host:5601 is completely empty. So I took a closer look at the output of filebeat setup -e and this is one of the last lines:

{
   "log.level":"error",
   "@timestamp":"2024-04-03T11:47:18.485+0200",
   "log.origin":{
      "function":"github.com/elastic/beats/v7/libbeat/cfgfile.(*Reloader).Load",
      "file.name":"cfgfile/reload.go",
      "file.line":255
   },
   "message":"Error loading configuration files: 1 error: Error creating runner from config: error getting filesets for module suricata: open /usr/share/filebeat/module/suricata: no such file or directory",
   "service.name":"filebeat",
   "ecs.version":"1.6.0"
}

So it seems like either the fileset config is wrong, or I'm missing the suricata module for filebeat.

However, filebeat modules enable suricata returns Module suricata is already enabled. So in my untrained head that means that filebeat knows what the suricata module is, even though it can't load it during setup.

What would be the best approch to debug this issue?
Is there any config files you guys need in order to dig into this?

Thanks in advance everyone and have an amazing day! :slight_smile:

Regards,
Jamo

2

Hello Jamo,

By reading the error message we can see that filebeat is looking for files under the following path: /usr/share/filebeat/module/suricata

I've check on my install on a debian machine and i confirm that the directory exists.

How did you install filebeat?

Thanks

3

Hi @metie ,

thanks for your reply!

That's good to know :thinking:

I've followed the official guide at Filebeat quick start: installation and configuration | Filebeat Reference [8.13] | Elastic

image
image860×235 8.45 KB

I think I might just reinstall filebeat at this point.

4

Hello @Jamo ,

I've check the reference filebeat.yml. While I did not validate this through functional configuration, it seems that you need to set the paths section to point to where you installed filebeat.

https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-reference-yml.html

https://www.elastic.co/guide/en/beats/filebeat/current/configuration-path.html

You can also validate that you have the correct path first by specifying the --path.home arguments to filebeat.

As an example, it should look like this: filebeat --path.home '/home/filebeat setup -e

Here is the reference to the command line argument specification

https://www.elastic.co/guide/en/beats/filebeat/current/command-line-options.html

Thanks!

5

Hi @metie

Thanks again for your reply!

sudo find / -name "filebeat" gives me just one result at /etc/filebeat, so I'm pretty sure that's the installation location of filebeat.

According to that, I than tried to run sudo filebeat setup --path.home '/etc/filebeat' -e

Unfortunately I'm still getting the same error message.

Error creating runner from config: error getting filesets for module suricata: open /usr/share/filebeat/module/suricata: no such file or directory","service.name":"filebeat","ecs.version":"1.6.0"

I just don't get why there's no suricata folder at /usr/share/filebeat/module. It seems like some step of the installation failed.

I only have the following modules under /usr/share/filebeat/module:

user@host:/usr/share/filebeat/module$ ls -l
drwxr-xr-x 4 root root 4096  3. Apr 01:19 apache
drwxr-xr-x 3 root root 4096  3. Apr 01:19 auditd
drwxr-xr-x 7 root root 4096  3. Apr 01:19 elasticsearch
drwxr-xr-x 3 root root 4096  3. Apr 01:19 haproxy
drwxr-xr-x 5 root root 4096  3. Apr 01:19 icinga
drwxr-xr-x 4 root root 4096  3. Apr 01:19 iis
drwxr-xr-x 3 root root 4096  3. Apr 01:19 kafka
drwxr-xr-x 4 root root 4096  3. Apr 01:19 kibana
drwxr-xr-x 4 root root 4096  3. Apr 01:19 logstash
drwxr-xr-x 3 root root 4096  3. Apr 01:19 mongodb
drwxr-xr-x 4 root root 4096  3. Apr 01:19 mysql
drwxr-xr-x 3 root root 4096  3. Apr 01:19 nats
drwxr-xr-x 5 root root 4096  3. Apr 01:19 nginx
drwxr-xr-x 3 root root 4096  3. Apr 01:19 osquery
drwxr-xr-x 3 root root 4096  3. Apr 01:19 pensando
drwxr-xr-x 3 root root 4096  3. Apr 01:19 postgresql
drwxr-xr-x 4 root root 4096  3. Apr 01:19 redis
drwxr-xr-x 3 root root 4096  3. Apr 01:19 santa
drwxr-xr-x 4 root root 4096  3. Apr 01:19 system
drwxr-xr-x 3 root root 4096  3. Apr 01:19 traefik
Entire Log output after running filebeat setup 
{
   "log.level":"info",
   "@timestamp":"2024-04-22T12:08:20.129+0200",
   "log.origin":{
      "function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).configure",
      "file.name":"instance/beat.go",
      "file.line":811
   },
   "message":"Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]",
   "service.name":"filebeat",
   "ecs.version":"1.6.0"
}{
   "log.level":"info",
   "@timestamp":"2024-04-22T12:08:20.129+0200",
   "log.origin":{
      "function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).configure",
      "file.name":"instance/beat.go",
      "file.line":819
   },
   "message":"Beat ID: 1e282e6b-e92c-4556-8455-cfea5f78a61b",
   "service.name":"filebeat",
   "ecs.version":"1.6.0"
}{
   "log.level":"info",
   "@timestamp":"2024-04-22T12:08:20.134+0200",
   "log.logger":"beat",
   "log.origin":{
      "function":"github.com/elastic/beats/v7/libbeat/cmd/instance.logSystemInfo",
      "file.name":"instance/beat.go",
      "file.line":1365
   },
   "message":"Beat info",
   "service.name":"filebeat",
   "system_info":{
      "beat":{
         "path":{
            "config":"/etc/filebeat",
            "data":"/var/lib/filebeat",
            "home":"/usr/share/filebeat",
            "logs":"/var/log/filebeat"
         },
         "type":"filebeat",
         "uuid":"1e282e6b-e92c-4556-8455-cfea5f78a61b"
      },
      "ecs.version":"1.6.0"
   }
}{
   "log.level":"info",
   "@timestamp":"2024-04-22T12:08:20.134+0200",
   "log.logger":"beat",
   "log.origin":{
      "function":"github.com/elastic/beats/v7/libbeat/cmd/instance.logSystemInfo",
      "file.name":"instance/beat.go",
      "file.line":1374
   },
   "message":"Build info",
   "service.name":"filebeat",
   "system_info":{
      "build":{
         "commit":"e9e462d71bdcd33a84d7f51753a116b5d418938f",
         "libbeat":"8.13.1",
         "time":"2024-03-27T15:39:08.000Z",
         "version":"8.13.1"
      },
      "ecs.version":"1.6.0"
   }
}{
   "log.level":"info",
   "@timestamp":"2024-04-22T12:08:20.134+0200",
   "log.logger":"beat",
   "log.origin":{
      "function":"github.com/elastic/beats/v7/libbeat/cmd/instance.logSystemInfo",
      "file.name":"instance/beat.go",
      "file.line":1377
   },
   "message":"Go runtime info",
   "service.name":"filebeat",
   "system_info":{
      "go":{
         "os":"linux",
         "arch":"amd64",
         "max_procs":6,
         "version":"go1.21.8"
      },
      "ecs.version":"1.6.0"
   }
}{
   "log.level":"info",
   "@timestamp":"2024-04-22T12:08:20.137+0200",
   "log.logger":"beat",
   "log.origin":{
      "function":"github.com/elastic/beats/v7/libbeat/cmd/instance.logSystemInfo",
      "file.name":"instance/beat.go",
      "file.line":1383
   },
   "message":"Host info",
   "service.name":"filebeat",
   "system_info":{
      "host":{
         "architecture":"x86_64",
         "boot_time":"2024-04-03T00:51:32+02:00",
         "containerized":false,
         "name":"bag-trace-suricata",
         "ip":[
            "127.0.0.1",
            "192.168.200.203"
         ],
         "kernel_version":"6.1.0-18-amd64",
         "mac":[
            "e6:a5:8e:c6:b3:59"
         ],
         "os":{
            "type":"linux",
            "family":"debian",
            "platform":"debian",
            "name":"Debian GNU/Linux",
            "version":"12 (bookworm)",
            "major":12,
            "minor":0,
            "patch":0,
            "codename":"bookworm"
         },
         "timezone":"CEST",
         "timezone_offset_sec":7200,
         "id":"6ac20534d9d04d2fa842b346f5c3b099"
      },
      "ecs.version":"1.6.0"
   }
}{
   "log.level":"info",
   "@timestamp":"2024-04-22T12:08:20.138+0200",
   "log.logger":"beat",
   "log.origin":{
      "function":"github.com/elastic/beats/v7/libbeat/cmd/instance.logSystemInfo",
      "file.name":"instance/beat.go",
      "file.line":1412
   },
   "message":"Process info",
   "service.name":"filebeat",
   "system_info":{
      "process":{
         "capabilities":{
            "inheritable":null,
            "permitted":[
               "chown",
               "dac_override",
               "dac_read_search",
               "fowner",
               "fsetid",
               "kill",
               "setgid",
               "setuid",
               "setpcap",
               "linux_immutable",
               "net_bind_service",
               "net_broadcast",
               "net_admin",
               "net_raw",
               "ipc_lock",
               "ipc_owner",
               "sys_module",
               "sys_rawio",
               "sys_chroot",
               "sys_ptrace",
               "sys_pacct",
               "sys_admin",
               "sys_boot",
               "sys_nice",
               "sys_resource",
               "sys_time",
               "sys_tty_config",
               "mknod",
               "lease",
               "audit_write",
               "audit_control",
               "setfcap",
               "mac_override",
               "mac_admin",
               "syslog",
               "wake_alarm",
               "block_suspend",
               "audit_read",
               "perfmon",
               "bpf",
               "checkpoint_restore"
            ],
            "effective":[
               "chown",
               "dac_override",
               "dac_read_search",
               "fowner",
               "fsetid",
               "kill",
               "setgid",
               "setuid",
               "setpcap",
               "linux_immutable",
               "net_bind_service",
               "net_broadcast",
               "net_admin",
               "net_raw",
               "ipc_lock",
               "ipc_owner",
               "sys_module",
               "sys_rawio",
               "sys_chroot",
               "sys_ptrace",
               "sys_pacct",
               "sys_admin",
               "sys_boot",
               "sys_nice",
               "sys_resource",
               "sys_time",
               "sys_tty_config",
               "mknod",
               "lease",
               "audit_write",
               "audit_control",
               "setfcap",
               "mac_override",
               "mac_admin",
               "syslog",
               "wake_alarm",
               "block_suspend",
               "audit_read",
               "perfmon",
               "bpf",
               "checkpoint_restore"
            ],
            "bounding":[
               "chown",
               "dac_override",
               "dac_read_search",
               "fowner",
               "fsetid",
               "kill",
               "setgid",
               "setuid",
               "setpcap",
               "linux_immutable",
               "net_bind_service",
               "net_broadcast",
               "net_admin",
               "net_raw",
               "ipc_lock",
               "ipc_owner",
               "sys_module",
               "sys_rawio",
               "sys_chroot",
               "sys_ptrace",
               "sys_pacct",
               "sys_admin",
               "sys_boot",
               "sys_nice",
               "sys_resource",
               "sys_time",
               "sys_tty_config",
               "mknod",
               "lease",
               "audit_write",
               "audit_control",
               "setfcap",
               "mac_override",
               "mac_admin",
               "syslog",
               "wake_alarm",
               "block_suspend",
               "audit_read",
               "perfmon",
               "bpf",
               "checkpoint_restore"
            ],
            "ambient":null
         },
         "cwd":"/home/bag",
         "exe":"/usr/share/filebeat/bin/filebeat",
         "name":"filebeat",
         "pid":127059,
         "ppid":127058,
         "seccomp":{
            "mode":"disabled",
            "no_new_privs":false
         },
         "start_time":"2024-04-22T12:08:19.560+0200"
      },
      "ecs.version":"1.6.0"
   }
}{
   "log.level":"info",
   "@timestamp":"2024-04-22T12:08:20.139+0200",
   "log.origin":{
      "function":"github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).createBeater",
      "file.name":"instance/beat.go",
      "file.line":334
   },
   "message":"Setup Beat: filebeat; Version: 8.13.1",
   "service.name":"filebeat",
   "ecs.version":"1.6.0"
}{
   "log.level":"info",
   "@timestamp":"2024-04-22T12:08:20.146+0200",
   "log.logger":"elasticsearch",
   "log.origin":{
      "function":"github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.makeES",
      "file.name":"elasticsearch/elasticsearch.go",
      "file.line":63
   },
   "message":"Applying performance preset 'balanced': {\n  \"bulk_max_size\": 1600,\n  \"compression_level\": 1,\n  \"idle_connection_timeout\": \"3s\",\n  \"queue\": {\n    \"mem\": {\n      \"events\": 3200,\n      \"flush\": {\n        \"min_events\": 1600,\n        \"timeout\": \"10s\"\n      }\n    }\n  },\n  \"worker\": 1\n}",
   "service.name":"filebeat",
   "ecs.version":"1.6.0"
}{
   "log.level":"warn",
   "@timestamp":"2024-04-22T12:08:20.146+0200",
   "log.logger":"elasticsearch",
   "log.origin":{
      "function":"github.com/elastic/beats/v7/libbeat/outputs/elasticsearch.makeES",
      "file.name":"elasticsearch/elasticsearch.go",
      "file.line":66
   },
   "message":"Performance preset 'balanced' overrides user setting for field 'bulk_max_size'",
   "service.name":"filebeat",
   "ecs.version":"1.6.0"
}{
   "log.level":"info",
   "@timestamp":"2024-04-22T12:08:20.146+0200",
   "log.logger":"esclientleg",
   "log.origin":{
      "function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.NewConnection",
      "file.name":"eslegclient/connection.go",
      "file.line":122
   },
   "message":"elasticsearch url: https://127.0.0.1:9200",
   "service.name":"filebeat",
   "ecs.version":"1.6.0"
}{
   "log.level":"warn",
   "@timestamp":"2024-04-22T12:08:20.147+0200",
   "log.logger":"tls",
   "log.origin":{
      "function":"github.com/elastic/elastic-agent-libs/transport/tlscommon.(*TLSConfig).ToConfig",
      "file.name":"tlscommon/tls_config.go",
      "file.line":107
   },
   "message":"SSL/TLS verifications disabled.",
   "service.name":"filebeat",
   "ecs.version":"1.6.0"
}{
   "log.level":"info",
   "@timestamp":"2024-04-22T12:08:20.147+0200",
   "log.logger":"publisher",
   "log.origin":{
      "function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.LoadWithSettings",
      "file.name":"pipeline/module.go",
      "file.line":105
   },
   "message":"Beat name: bag-trace-suricata",
   "service.name":"filebeat",
   "ecs.version":"1.6.0"
}{
   "log.level":"info",
   "@timestamp":"2024-04-22T12:08:20.149+0200",
   "log.logger":"modules",
   "log.origin":{
      "function":"github.com/elastic/beats/v7/filebeat/fileset.newModuleRegistry",
      "file.name":"fileset/modules.go",
      "file.line":135
   },
   "message":"Enabled modules/filesets: system (auth), system (syslog)",
   "service.name":"filebeat",
   "ecs.version":"1.6.0"
}{
   "log.level":"info",
   "@timestamp":"2024-04-22T12:08:20.151+0200",
   "log.logger":"esclientleg",
   "log.origin":{
      "function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.NewConnection",
      "file.name":"eslegclient/connection.go",
      "file.line":122
   },
   "message":"elasticsearch url: https://127.0.0.1:9200",
   "service.name":"filebeat",
   "ecs.version":"1.6.0"
}{
   "log.level":"warn",
   "@timestamp":"2024-04-22T12:08:20.151+0200",
   "log.logger":"tls",
   "log.origin":{
      "function":"github.com/elastic/elastic-agent-libs/transport/tlscommon.(*TLSConfig).ToConfig",
      "file.name":"tlscommon/tls_config.go",
      "file.line":107
   },
   "message":"SSL/TLS verifications disabled.",
   "service.name":"filebeat",
   "ecs.version":"1.6.0"
}{
   "log.level":"warn",
   "@timestamp":"2024-04-22T12:08:20.151+0200",
   "log.logger":"tls",
   "log.origin":{
      "function":"github.com/elastic/elastic-agent-libs/transport/tlscommon.(*TLSConfig).ToConfig",
      "file.name":"tlscommon/tls_config.go",
      "file.line":107
   },
   "message":"SSL/TLS verifications disabled.",
   "service.name":"filebeat",
   "ecs.version":"1.6.0"
}{
   "log.level":"info",
   "@timestamp":"2024-04-22T12:08:20.187+0200",
   "log.logger":"esclientleg",
   "log.origin":{
      "function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Ping",
      "file.name":"eslegclient/connection.go",
      "file.line":304
   },
   "message":"Attempting to connect to Elasticsearch version 8.13.1 (default)",
   "service.name":"filebeat",
   "ecs.version":"1.6.0"
}{
   "log.level":"info",
   "@timestamp":"2024-04-22T12:08:20.190+0200",
   "log.logger":"modules",
   "log.origin":{
      "function":"github.com/elastic/beats/v7/filebeat/fileset.newModuleRegistry",
      "file.name":"fileset/modules.go",
      "file.line":135
   },
   "message":"Enabled modules/filesets: elasticsearch (server)",
   "service.name":"filebeat",
   "ecs.version":"1.6.0"
}{
   "log.level":"info",
   "@timestamp":"2024-04-22T12:08:20.191+0200",
   "log.logger":"esclientleg",
   "log.origin":{
      "function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.NewConnection",
      "file.name":"eslegclient/connection.go",
      "file.line":122
   },
   "message":"elasticsearch url: https://127.0.0.1:9200",
   "service.name":"filebeat",
   "ecs.version":"1.6.0"
}{
   "log.level":"warn",
   "@timestamp":"2024-04-22T12:08:20.191+0200",
   "log.logger":"tls",
   "log.origin":{
      "function":"github.com/elastic/elastic-agent-libs/transport/tlscommon.(*TLSConfig).ToConfig",
      "file.name":"tlscommon/tls_config.go",
      "file.line":107
   },
   "message":"SSL/TLS verifications disabled.",
   "service.name":"filebeat",
   "ecs.version":"1.6.0"
}{
   "log.level":"warn",
   "@timestamp":"2024-04-22T12:08:20.191+0200",
   "log.logger":"tls",
   "log.origin":{
      "function":"github.com/elastic/elastic-agent-libs/transport/tlscommon.(*TLSConfig).ToConfig",
      "file.name":"tlscommon/tls_config.go",
      "file.line":107
   },
   "message":"SSL/TLS verifications disabled.",
   "service.name":"filebeat",
   "ecs.version":"1.6.0"
}{
   "log.level":"info",
   "@timestamp":"2024-04-22T12:08:20.226+0200",
   "log.logger":"esclientleg",
   "log.origin":{
      "function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Ping",
      "file.name":"eslegclient/connection.go",
      "file.line":304
   },
   "message":"Attempting to connect to Elasticsearch version 8.13.1 (default)",
   "service.name":"filebeat",
   "ecs.version":"1.6.0"
}{
   "log.level":"info",
   "@timestamp":"2024-04-22T12:08:20.233+0200",
   "log.logger":"modules",
   "log.origin":{
      "function":"github.com/elastic/beats/v7/filebeat/fileset.LoadPipeline",
      "file.name":"fileset/pipelines.go",
      "file.line":135
   },
   "message":"Elasticsearch pipeline loaded.",
   "service.name":"filebeat",
   "pipeline":"filebeat-8.13.1-elasticsearch-server-pipeline",
   "ecs.version":"1.6.0"
}{
   "log.level":"info",
   "@timestamp":"2024-04-22T12:08:20.234+0200",
   "log.logger":"modules",
   "log.origin":{
      "function":"github.com/elastic/beats/v7/filebeat/fileset.LoadPipeline",
      "file.name":"fileset/pipelines.go",
      "file.line":135
   },
   "message":"Elasticsearch pipeline loaded.",
   "service.name":"filebeat",
   "pipeline":"filebeat-8.13.1-elasticsearch-server-pipeline-plaintext",
   "ecs.version":"1.6.0"
}{
   "log.level":"info",
   "@timestamp":"2024-04-22T12:08:20.236+0200",
   "log.logger":"modules",
   "log.origin":{
      "function":"github.com/elastic/beats/v7/filebeat/fileset.LoadPipeline",
      "file.name":"fileset/pipelines.go",
      "file.line":135
   },
   "message":"Elasticsearch pipeline loaded.",
   "service.name":"filebeat",
   "pipeline":"filebeat-8.13.1-elasticsearch-server-pipeline-json",
   "ecs.version":"1.6.0"
}{
   "log.level":"info",
   "@timestamp":"2024-04-22T12:08:20.238+0200",
   "log.logger":"modules",
   "log.origin":{
      "function":"github.com/elastic/beats/v7/filebeat/fileset.LoadPipeline",
      "file.name":"fileset/pipelines.go",
      "file.line":135
   },
   "message":"Elasticsearch pipeline loaded.",
   "service.name":"filebeat",
   "pipeline":"filebeat-8.13.1-elasticsearch-server-pipeline-json-7",
   "ecs.version":"1.6.0"
}{
   "log.level":"info",
   "@timestamp":"2024-04-22T12:08:20.240+0200",
   "log.logger":"modules",
   "log.origin":{
      "function":"github.com/elastic/beats/v7/filebeat/fileset.LoadPipeline",
      "file.name":"fileset/pipelines.go",
      "file.line":135
   },
   "message":"Elasticsearch pipeline loaded.",
   "service.name":"filebeat",
   "pipeline":"filebeat-8.13.1-elasticsearch-server-pipeline-json-8",
   "ecs.version":"1.6.0"
}{
   "log.level":"info",
   "@timestamp":"2024-04-22T12:08:20.241+0200",
   "log.logger":"modules",
   "log.origin":{
      "function":"github.com/elastic/beats/v7/filebeat/fileset.newModuleRegistry",
      "file.name":"fileset/modules.go",
      "file.line":135
   },
   "message":"Enabled modules/filesets: kibana (log)",
   "service.name":"filebeat",
   "ecs.version":"1.6.0"
}{
   "log.level":"info",
   "@timestamp":"2024-04-22T12:08:20.241+0200",
   "log.logger":"esclientleg",
   "log.origin":{
      "function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.NewConnection",
      "file.name":"eslegclient/connection.go",
      "file.line":122
   },
   "message":"elasticsearch url: https://127.0.0.1:9200",
   "service.name":"filebeat",
   "ecs.version":"1.6.0"
}{
   "log.level":"warn",
   "@timestamp":"2024-04-22T12:08:20.241+0200",
   "log.logger":"tls",
   "log.origin":{
      "function":"github.com/elastic/elastic-agent-libs/transport/tlscommon.(*TLSConfig).ToConfig",
      "file.name":"tlscommon/tls_config.go",
      "file.line":107
   },
   "message":"SSL/TLS verifications disabled.",
   "service.name":"filebeat",
   "ecs.version":"1.6.0"
}{
   "log.level":"warn",
   "@timestamp":"2024-04-22T12:08:20.241+0200",
   "log.logger":"tls",
   "log.origin":{
      "function":"github.com/elastic/elastic-agent-libs/transport/tlscommon.(*TLSConfig).ToConfig",
      "file.name":"tlscommon/tls_config.go",
      "file.line":107
   },
   "message":"SSL/TLS verifications disabled.",
   "service.name":"filebeat",
   "ecs.version":"1.6.0"
}{
   "log.level":"info",
   "@timestamp":"2024-04-22T12:08:20.278+0200",
   "log.logger":"esclientleg",
   "log.origin":{
      "function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Ping",
      "file.name":"eslegclient/connection.go",
      "file.line":304
   },
   "message":"Attempting to connect to Elasticsearch version 8.13.1 (default)",
   "service.name":"filebeat",
   "ecs.version":"1.6.0"
}{
   "log.level":"info",
   "@timestamp":"2024-04-22T12:08:20.285+0200",
   "log.logger":"modules",
   "log.origin":{
      "function":"github.com/elastic/beats/v7/filebeat/fileset.LoadPipeline",
      "file.name":"fileset/pipelines.go",
      "file.line":135
   },
   "message":"Elasticsearch pipeline loaded.",
   "service.name":"filebeat",
   "pipeline":"filebeat-8.13.1-kibana-log-pipeline",
   "ecs.version":"1.6.0"
}{
   "log.level":"info",
   "@timestamp":"2024-04-22T12:08:20.287+0200",
   "log.logger":"modules",
   "log.origin":{
      "function":"github.com/elastic/beats/v7/filebeat/fileset.LoadPipeline",
      "file.name":"fileset/pipelines.go",
      "file.line":135
   },
   "message":"Elasticsearch pipeline loaded.",
   "service.name":"filebeat",
   "pipeline":"filebeat-8.13.1-kibana-log-pipeline-7",
   "ecs.version":"1.6.0"
}{
   "log.level":"info",
   "@timestamp":"2024-04-22T12:08:20.289+0200",
   "log.logger":"modules",
   "log.origin":{
      "function":"github.com/elastic/beats/v7/filebeat/fileset.LoadPipeline",
      "file.name":"fileset/pipelines.go",
      "file.line":135
   },
   "message":"Elasticsearch pipeline loaded.",
   "service.name":"filebeat",
   "pipeline":"filebeat-8.13.1-kibana-log-pipeline-ecs",
   "ecs.version":"1.6.0"
}{
   "log.level":"error",
   "@timestamp":"2024-04-22T12:08:20.289+0200",
   "log.logger":"load",
   "log.origin":{
      "function":"github.com/elastic/beats/v7/libbeat/cfgfile.(*RunnerList).Reload",
      "file.name":"cfgfile/list.go",
      "file.line":138
   },
   "message":"Error creating runner from config: error getting filesets for module suricata: open /usr/share/filebeat/module/suricata: no such file or directory",
   "service.name":"filebeat",
   "ecs.version":"1.6.0"
}{
   "log.level":"info",
   "@timestamp":"2024-04-22T12:08:20.291+0200",
   "log.logger":"modules",
   "log.origin":{
      "function":"github.com/elastic/beats/v7/filebeat/fileset.newModuleRegistry",
      "file.name":"fileset/modules.go",
      "file.line":135
   },
   "message":"Enabled modules/filesets: system (syslog), system (auth)",
   "service.name":"filebeat",
   "ecs.version":"1.6.0"
}{
   "log.level":"info",
   "@timestamp":"2024-04-22T12:08:20.292+0200",
   "log.logger":"esclientleg",
   "log.origin":{
      "function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.NewConnection",
      "file.name":"eslegclient/connection.go",
      "file.line":122
   },
   "message":"elasticsearch url: https://127.0.0.1:9200",
   "service.name":"filebeat",
   "ecs.version":"1.6.0"
}{
   "log.level":"warn",
   "@timestamp":"2024-04-22T12:08:20.292+0200",
   "log.logger":"tls",
   "log.origin":{
      "function":"github.com/elastic/elastic-agent-libs/transport/tlscommon.(*TLSConfig).ToConfig",
      "file.name":"tlscommon/tls_config.go",
      "file.line":107
   },
   "message":"SSL/TLS verifications disabled.",
   "service.name":"filebeat",
   "ecs.version":"1.6.0"
}{
   "log.level":"warn",
   "@timestamp":"2024-04-22T12:08:20.293+0200",
   "log.logger":"tls",
   "log.origin":{
      "function":"github.com/elastic/elastic-agent-libs/transport/tlscommon.(*TLSConfig).ToConfig",
      "file.name":"tlscommon/tls_config.go",
      "file.line":107
   },
   "message":"SSL/TLS verifications disabled.",
   "service.name":"filebeat",
   "ecs.version":"1.6.0"
}{
   "log.level":"info",
   "@timestamp":"2024-04-22T12:08:20.331+0200",
   "log.logger":"esclientleg",
   "log.origin":{
      "function":"github.com/elastic/beats/v7/libbeat/esleg/eslegclient.(*Connection).Ping",
      "file.name":"eslegclient/connection.go",
      "file.line":304
   },
   "message":"Attempting to connect to Elasticsearch version 8.13.1 (default)",
   "service.name":"filebeat",
   "ecs.version":"1.6.0"
}{
   "log.level":"info",
   "@timestamp":"2024-04-22T12:08:20.335+0200",
   "log.logger":"modules",
   "log.origin":{
      "function":"github.com/elastic/beats/v7/filebeat/fileset.LoadPipeline",
      "file.name":"fileset/pipelines.go",
      "file.line":135
   },
   "message":"Elasticsearch pipeline loaded.",
   "service.name":"filebeat",
   "pipeline":"filebeat-8.13.1-system-syslog-pipeline",
   "ecs.version":"1.6.0"
}{
   "log.level":"info",
   "@timestamp":"2024-04-22T12:08:20.340+0200",
   "log.logger":"modules",
   "log.origin":{
      "function":"github.com/elastic/beats/v7/filebeat/fileset.LoadPipeline",
      "file.name":"fileset/pipelines.go",
      "file.line":135
   },
   "message":"Elasticsearch pipeline loaded.",
   "service.name":"filebeat",
   "pipeline":"filebeat-8.13.1-system-auth-pipeline",
   "ecs.version":"1.6.0"
}{
   "log.level":"error",
   "@timestamp":"2024-04-22T12:08:20.340+0200",
   "log.origin":{
      "function":"github.com/elastic/beats/v7/libbeat/cfgfile.(*Reloader).Load",
      "file.name":"cfgfile/reload.go",
      "file.line":255
   },
   "message":"Error loading configuration files: 1 error: Error creating runner from config: error getting filesets for module suricata: open /usr/share/filebeat/module/suricata: no such file or directory",
   "service.name":"filebeat",
   "ecs.version":"1.6.0"
}{
   "log.level":"info",
   "@timestamp":"2024-04-22T12:08:20.340+0200",
   "log.logger":"load",
   "log.origin":{
      "function":"github.com/elastic/beats/v7/libbeat/cfgfile.(*RunnerList).Stop",
      "file.name":"cfgfile/list.go",
      "file.line":188
   },
   "message":"Stopping 3 runners ...",
   "service.name":"filebeat",
   "ecs.version":"1.6.0"
}{
   "log.level":"info",
   "@timestamp":"2024-04-22T12:08:20.345+0200",
   "log.logger":"modules",
   "log.origin":{
      "function":"github.com/elastic/beats/v7/filebeat/fileset.LoadPipeline",
      "file.name":"fileset/pipelines.go",
      "file.line":135
   },
   "message":"Elasticsearch pipeline loaded.",
   "service.name":"filebeat",
   "pipeline":"filebeat-8.13.1-system-auth-pipeline",
   "ecs.version":"1.6.0"
}{
   "log.level":"info",
   "@timestamp":"2024-04-22T12:08:20.348+0200",
   "log.logger":"modules",
   "log.origin":{
      "function":"github.com/elastic/beats/v7/filebeat/fileset.LoadPipeline",
      "file.name":"fileset/pipelines.go",
      "file.line":135
   },
   "message":"Elasticsearch pipeline loaded.",
   "service.name":"filebeat",
   "pipeline":"filebeat-8.13.1-system-syslog-pipeline",
   "ecs.version":"1.6.0"
}
6

Okay, so I've now re-installed filebeat with apt. The directory /usr/share/filebeat/module/suricata now exists and isn't empty. Nice!

However, when I try to run the setup, I now receive the following error:
Exiting: module suricata is configured but has no enabled filesets
The same problem seems to be present for the system module.
Exiting: module system is configured but has no enabled filesets

Here's my config files:

/etc/filebeat/modules.d/suricata.yml 
- module: suricata
  eve:
    enabled: true
    var.paths: ["/var/log/suricata/eve.json"]
/etc/filebeat/modules.d/system.yml 
- module: system
  syslog:
    enabled: true
  auth:
    enabled: true

I can't spot any error here.

Setting up Elastic really drives me crazy now! You need to install Elasticsearch as an engine, Kibana for the dashboards (if I understood right), Filebeat for allowing Elastic to read files as input, and the Suricata module to tell Filebeat how to interpret the eve.json. It's just so much hastle to go through just to make a JSON readable for, in my case, Grafana, and the setup is just pure pain.
Has nothing to do with my thread, I know. But I just had to let it out.

7

Based on this Reddit post I've tried to setup the system module via sudo filebeat setup --pipelines --modules system -M "system.syslog.enabled=true" -M "system.auth.enabled=true"

For me this looks like the variables for system.auth.enabled and system.syslog.enabled, which are normally already configured within the system.yml, are being overridden.

I've also added the -e parameter to make sure that no error passes by unseen.
The setup finishes within a few seconds, and there are no errors to be found. At first, this was quite promising for me!

However, when I run sudo filebeat setup --modules system -e I get error: Unable to hash given config: missing field accessing '0.vpcflow' (source:'/etc/filebeat/modules.d/gcp.yml.disabled'), which makes even less sense considering that gcp is enabled and should therefore be ignored, shouldn't it?

When I try to run the same command for suricata instead of system I get the same error.

The more I try to debug this, the more frustrated I get.

8

I tried to have a look at the GCP Docs, and look at this:

photo_5276076245068667488_x
photo_5276076245068667488_x716×129 11.4 KB

photo_5276076245068667489_y
photo_5276076245068667489_y1280×449 65.4 KB

That has the be the worst joke I've seen today.

9

Aaaaand here I am again, continuing my monologue.

I've now moved my settings from modules.d/suricata.yml and modules.d/system.yml into filebeat/filebeat.yml. That allowed me to run the setups for both system and suricata without any errors. I also ran the full setup via sudo filebeat setup -e and got no errors.

However, the Kibana dashboard still shows nothing.

image
image1920×911 60.5 KB

The discover page is also completely empty

image
image1919×658 77.3 KB

I'm absolutely out of ideas and patience.

10

@Jamo If you get everything right(As it seems you have now), at this point, it might be a time frame issue.

As you can see in the screenshot you're looking at the 15 past minutes. This might need an adjustement.

Can you look at the timestamp of your eve.json and confirm that the time frame is overlaping this?

Also can you show the index management pane so you can confirm that there is data ingested and which index is used. The index management menu, is in stack management -> index management. You might have to include hidden indices.