[BUG] Bug in logs related to Kafka connection

Elastic Stack Beats
1

Good afternoon everyone,

I think that I have found a bug in the logs of Filebeat, version 7.12.0.

I have a Kafka machine in a datacenter, let's call it Datacenter A, and I setup the Filebeat in another datacenter, in this case the Datacenter B.

I configured my Filebeat with Kafka as output, and I was wondering why I didn't receive any event in my Kafka topic, when Filebeat said that everything was okay and that it was sending events.

Well, the problem is that I didn't have opened the port 6397 for the Kafka host, so basically when I did a telnet call I discovered that it was impossible to connect. But when I go to my Filebeat logs, it says the following:

2021-04-08T18:27:07.627+0200    INFO    [publisher_pipeline_output]     pipeline/output.go:143  Connecting to kafka(5X.XX.XX.XX:6397)
2021-04-08T18:27:07.627+0200    INFO    [publisher_pipeline_output]     pipeline/output.go:151  Connection to kafka(5X.XX.XX.XX:6397) established

So I suppose there is some sort of error in the logging, since it's literally impossible that Filebeat has established connection with Kafka in that host and port.

Of course, after opening the port in the firewall, it started working properly and sending the events.

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

3

Hi @dacamposol,

This log message is misleading, it does not mean that the actual connection was created, it just means that the worker has been correctly setup and is read to send data. The real connection to the Kafka server will only be established when the first event is sent.

With Filebeat you can use the ./filebeat test output command to test the connection to an output, this will create a connection and return whether it was successful or not.

There is some more information on the GH issue: Beats misreport established Kafka connection · Issue #34177 · elastic/beats · GitHub