Good afternoon everyone,
I think that I have found a bug in the logs of Filebeat, version 7.12.0.
I have a Kafka machine in a datacenter, let's call it Datacenter A, and I setup the Filebeat in another datacenter, in this case the Datacenter B.
I configured my Filebeat with Kafka as output, and I was wondering why I didn't receive any event in my Kafka topic, when Filebeat said that everything was okay and that it was sending events.
Well, the problem is that I didn't have opened the port 6397 for the Kafka host, so basically when I did a telnet call I discovered that it was impossible to connect. But when I go to my Filebeat logs, it says the following:
2021-04-08T18:27:07.627+0200 INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to kafka(5X.XX.XX.XX:6397) 2021-04-08T18:27:07.627+0200 INFO [publisher_pipeline_output] pipeline/output.go:151 Connection to kafka(5X.XX.XX.XX:6397) established
So I suppose there is some sort of error in the logging, since it's literally impossible that Filebeat has established connection with Kafka in that host and port.
Of course, after opening the port in the firewall, it started working properly and sending the events.