I have setup a second instance of Filebeat to test the config on another ELK stack server using this command:
/usr/share/filebeat/bin/filebeat -c /etc/filebeat/filebeat-test.yml -path.data /var/lib/filebeat/tempdat-path.logs /var/log/filebeat-test -E logging.level=debug
The thing I have observed through tcpdump is that it is not sending any traffic through the network interface at all. Here is the debug log from filebeat-test:
2020-05-20T08:21:30.684+0200 WARN beater/filebeat.go:261 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020-05-20T08:21:30.684+0200 INFO crawler/crawler.go:48 Loading Prospectors: 0
2020-05-20T08:21:30.684+0200 DEBUG [cfgfile] cfgfile/reload.go:95 Checking module configs from: /usr/share/filebeat/bin/modules.d/*.yml
2020-05-20T08:21:30.684+0200 DEBUG [cfgfile] cfgfile/reload.go:109 Number of module configs found: 0
2020-05-20T08:21:30.684+0200 INFO crawler/crawler.go:82 Loading and starting Prospectors completed. Enabled prospectors: 0
2020-05-20T08:21:30.684+0200 DEBUG [registrar] registrar/registrar.go:152 Starting Registrar
2020-05-20T08:21:30.684+0200 INFO cfgfile/reload.go:127 Config reloader started
2020-05-20T08:21:30.685+0200 DEBUG [cfgfile] cfgfile/reload.go:151 Scan for new config files
2020-05-20T08:21:30.685+0200 DEBUG [cfgfile] cfgfile/reload.go:170 Number of module configs found: 0
2020-05-20T08:21:30.685+0200 INFO cfgfile/reload.go:219 Loading of config files completed.
2020-05-20T08:22:00.684+0200 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":10,"time":16},"total":{"ticks":10,"time":24,"value":10},"user":{"ticks":0,"time":8}},"info":{"ephemeral_id":"e373a965-50c3-448a-a623-b86d2d705b59","uptime":{"ms":30010}},"memstats":{"gc_next":4194304,"memory_alloc":1481360,"memory_total":3143832,"rss":20623360}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"type":"logstash"},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":32},"load":{"1":4.82,"15":4.16,"5":4.28,"norm":{"1":0.1506,"15":0.13,"5":0.1338}}}}}}
2020-05-20T08:22:30.683+0200 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":20,"time":20},"total":{"ticks":20,"time":28,"value":20},"user":{"ticks":0,"time":8}},"info":{"ephemeral_id":"e373a965-50c3-448a-a623-b86d2d705b59","uptime":{"ms":60009}},"memstats":{"gc_next":4194304,"memory_alloc":1694032,"memory_total":3356504}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"load":{"1":5.15,"15":4.21,"5":4.43,"norm":{"1":0.1609,"15":0.1316,"5":0.1384}}}}}}
I have tested with telnet that communication between the filebeat server and the ELK stack server exists.
I also made a curl call on the ELK stack server and it shows no connections.
Any suggestions what could be the issue? or how I could test this in a different way?