Hello,
I have been on a journey trying to use the iptables module in filebeat 7.16.3. I suspected the distro I'm using was not installing something right, specifically missing the modules/iptables directory. I then tried to use precompiled binaries, and got this strange situation:
https://discuss.elastic.co/t/bizarre-error-bash-filebeat-no-such-file-or-directory
As such, I moved on to compiling from source. I downloaded this, which is apparently 7.16.3(?):
https://github.com/elastic/beats/archive/refs/heads/7.16.zip
Uncompressed it, and then did make -j8 (as there are 8 cores on that system). Things seemed to be going well. I edited up filebeat.yml to my needs (send data to my already functioning logstash docker, part of my already functioning ELK stack which is taking in many logs from many sources), and enabled the iptables module with this command:
./filebeat -c ./filebeat.yml enable modules iptables
And I was informed that it had been enabled. From here, I edited modules.d/iptables.yml to my needs, and then started filebeat with -e so that I can see the logs on STDERR:
./filebeat -c ./filebeat.yml -e
This gives me the following output and then it exits. I am sending to logstash instead of Elasticsearch, so the WARN lines about that can be ignored:
2022-02-07T16:40:32.444Z INFO instance/beat.go:686 Home path: [/root/beats-7.16/filebeat] Config path: [/root/beats-7.16/filebeat] Data path: [/root/beats-7.16/filebeat/data] Logs path: [/
root/beats-7.16/filebeat/logs] Hostfs Path: [/]
2022-02-07T16:40:32.444Z INFO instance/beat.go:694 Beat ID: 0cce485c-4f43-47ae-b872-cdbf664fe86e
2022-02-07T16:40:35.445Z WARN [add_cloud_metadata] add_cloud_metadata/provider_aws_ec2.go:79 read token request for getting IMDSv2 token returns empty: Put "http://169.254.169.254/la
test/api/token": context deadline exceeded (Client.Timeout exceeded while awaiting headers). No token in the metadata request will be used.
2022-02-07T16:40:35.447Z INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully installed
2022-02-07T16:40:35.447Z INFO [beat] instance/beat.go:1040 Beat info {"system_info": {"beat": {"path": {"config": "/root/beats-7.16/filebeat", "data": "/root/beats-7.16/filebeat/data
", "home": "/root/beats-7.16/filebeat", "logs": "/root/beats-7.16/filebeat/logs"}, "type": "filebeat", "uuid": "0cce485c-4f43-47ae-b872-cdbf664fe86e"}}}
2022-02-07T16:40:35.447Z INFO [beat] instance/beat.go:1049 Build info {"system_info": {"build": {"commit": "unknown", "libbeat": "7.16.4", "time": "1754-08-30T22:43:41.128Z", "version
": "7.16.4"}}}
2022-02-07T16:40:35.447Z INFO [beat] instance/beat.go:1052 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":8,"version":"go1.17.4"}}}
2022-02-07T16:40:35.448Z INFO [beat] instance/beat.go:1056 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2022-02-07T15:43:05Z","containerized":false,"name"
:"alpine-fb-test","ip":["127.0.0.1/8","::1/128","10.1.1.38/24","fe80::20c:29ff:fe46:9706/64"],"kernel_version":"5.15.16-0-virt","mac":["00:0c:29:46:97:06"],"os":{"type":"linux","family":"","platform":"
alpine","name":"Alpine Linux","version":"","major":0,"minor":0,"patch":0},"timezone":"UTC","timezone_offset_sec":0}}}
2022-02-07T16:40:35.448Z INFO [beat] instance/beat.go:1085 Process info {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_se
arch","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_p
trace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm",
"block_suspend","audit_read","38","39","40"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadca
st","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lea
se","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39","40"],"bounding":["chown","dac_override","dac_read_search","fowner","
fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacc
t","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend",
"audit_read","38","39","40"],"ambient":null}, "cwd": "/root/beats-7.16/filebeat", "exe": "/root/beats-7.16/filebeat/filebeat", "name": "filebeat", "pid": 11244, "ppid": 3811, "seccomp": {"mode":"filter
","no_new_privs":true}, "start_time": "2022-02-07T16:40:31.590Z"}}}
2022-02-07T16:40:35.448Z INFO instance/beat.go:328 Setup Beat: filebeat; Version: 7.16.4
2022-02-07T16:40:35.448Z INFO [publisher] pipeline/module.go:113 Beat name: alpine-fb-test
2022-02-07T16:40:35.448Z WARN beater/filebeat.go:202 Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If
you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.
2022-02-07T16:40:35.449Z INFO [monitoring] log/log.go:142 Starting metrics logging every 30s
2022-02-07T16:40:35.449Z INFO instance/beat.go:492 filebeat start running.
2022-02-07T16:40:35.449Z INFO memlog/store.go:119 Loading data file of '/root/beats-7.16/filebeat/data/registry/filebeat' succeeded. Active transaction id=0
2022-02-07T16:40:35.450Z INFO memlog/store.go:124 Finished loading transaction log file for '/root/beats-7.16/filebeat/data/registry/filebeat'. Active transaction id=85
2022-02-07T16:40:35.450Z WARN beater/filebeat.go:411 Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If
you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.
2022-02-07T16:40:35.450Z INFO [registrar] registrar/registrar.go:109 States Loaded from registrar: 0
2022-02-07T16:40:35.450Z INFO [crawler] beater/crawler.go:71 Loading Inputs: 1
2022-02-07T16:40:35.450Z INFO [crawler] beater/crawler.go:141 Starting input (ID: 2631682662219732373)
2022-02-07T16:40:35.450Z INFO beater/crawler.go:148 Stopping Crawler
2022-02-07T16:40:35.450Z INFO beater/crawler.go:158 Stopping 1 inputs
2022-02-07T16:40:35.450Z INFO [input.filestream] compat/compat.go:111 Input filestream starting {"id": "24859D7023295595"}
2022-02-07T16:40:35.450Z INFO [crawler] beater/crawler.go:163 Stopping input: 2631682662219732373
2022-02-07T16:40:35.450Z INFO [input.filestream] compat/compat.go:132 Input 'filestream' stopped {"id": "24859D7023295595"}
2022-02-07T16:40:35.450Z INFO beater/crawler.go:178 Crawler stopped
2022-02-07T16:40:35.450Z INFO [registrar] registrar/registrar.go:132 Stopping Registrar
2022-02-07T16:40:35.450Z INFO [registrar] registrar/registrar.go:166 Ending Registrar
2022-02-07T16:40:35.450Z INFO [registrar] registrar/registrar.go:137 Registrar stopped
2022-02-07T16:40:35.450Z INFO [file_watcher] filestream/fswatch.go:137 Start next scan
2022-02-07T16:40:35.451Z INFO [input.filestream] compat/compat.go:124 Input 'filestream' stopped {"id": "24859D7023295595"}
2022-02-07T16:40:35.453Z INFO [monitoring] log/log.go:192 Total metrics {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":30,"time":{"ms":39}},"total":{"ticks":50,"time":{"ms
":65},"value":0},"user":{"ticks":20,"time":{"ms":26}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":15},"info":{"ephemeral_id":"17a40c88-5ca0-4056-a4a7-99042bc6cb7c","uptime":{"ms":3024},"versio
n":"7.16.4"},"memstats":{"gc_next":9645344,"memory_alloc":6572024,"memory_sys":21054472,"memory_total":14163304,"rss":49483776},"runtime":{"goroutines":42}},"filebeat":{"events":{"active":0,"added":0,"
done":0},"harvester":{"closed":0,"open_files":0,"running":0,"skipped":0,"started":0},"input":{"log":{"files":{"renamed":0,"truncated":0}}}},"libbeat":{"config":{"module":{"running":0,"starts":0,"stops"
:0},"reloads":0,"scans":0},"output":{"events":{"acked":0,"active":0,"batches":0,"dropped":0,"duplicates":0,"failed":0,"toomany":0,"total":0},"read":{"bytes":0,"errors":0},"type":"logstash","write":{"by
tes":0,"errors":0}},"pipeline":{"clients":0,"events":{"active":0,"dropped":0,"failed":0,"filtered":0,"published":0,"retry":0,"total":0},"queue":{"acked":0,"max_events":4096}}},"registrar":{"states":{"c
leanup":0,"current":0,"update":0},"writes":{"fail":0,"success":0,"total":0}},"system":{"cpu":{"cores":8},"load":{"1":0,"15":0.04,"5":0,"norm":{"1":0,"15":0.005,"5":0}}}}}}
2022-02-07T16:40:35.453Z INFO [monitoring] log/log.go:193 Uptime: 3.025075951s
2022-02-07T16:40:35.453Z INFO [monitoring] log/log.go:160 Stopping metrics logging.
2022-02-07T16:40:35.453Z INFO instance/beat.go:498 filebeat stopped.
2022-02-07T16:40:35.453Z ERROR instance/beat.go:1015 Exiting: Failed to start crawler: creating module reloader failed: could not create module registry for filesets: error getting filesets
for module iptables: open /root/beats-7.16/filebeat/module/iptables: no such file or directory***
Exiting: Failed to start crawler: creating module reloader failed: could not create module registry for filesets: error getting filesets for module iptables: open /root/beats-7.16/filebeat/module/iptab
les: no such file or directory
As it says, /root/beats-7.16/filebeat/module/iptables does not exist. Yet /root/beats-7.16/filebeat/modules.d/iptables.yml does exist and came with the source code (initially as iptables.yml.disabled of course), which I edited to my needs.
So I don't understand: If I have enabled the iptables module, configured its .yml properly, and so on, why am I getting this error? Did I miss a step?
