These logs indicate the authentication did succeed:
2018-02-13T08:05:50.078+0100 INFO kafka/log.go:36 kafka message: [[Successful SASL handshake]]
2018-02-13T08:05:50.105+0100 INFO kafka/log.go:36 SASL authentication successful with broker
On first connect metricbeat tries to match the broker id with the actual hostname and port. For this metricbeat uses the kafka API GetMetadata call, to query the cluster state.
These message indicate the query was successfull (=> SASL authentication was successful):
2018-02-13T08:05:50.134+0100 DEBUG [kafka] kafka/broker.go:348 Try to match broker to: localhost:20160
2018-02-13T08:05:50.134+0100 DEBUG [kafka] kafka/broker.go:109 found matching broker localhost:20160 with id 4
2018-02-13T08:05:50.161+0100 INFO kafka/log.go:36 Closed connection to broker [[localhost:20160]]
The metricbeat kafka module does not execute the normal connection bootstrapping. Instead one should have one metricbeat instance per kafka host (or configure all hosts with the module). The state in kafka is very distributed between nodes. You need to connect to the correct broker in order to collect metrics.
At this time we have one metricbeat per kafka node and connect via localhost tho the "local" instance. I can try to configura all cluster nodes an see if it makes any diffenrce.
I may add our configuration worked until we activated the SASL Auth.
I changed my configuration to access all cluster nodes with the metricbeat instnace. The Result is the same. I didn't get any metrics back. The Log looks as reported previously, execp that there is an entry per server.
it looks like that the broker isn't registered or metricbeat can't get the id correct:
if b.id >= 0 {
Logger.Printf("Connected to broker at %s (registered as #%d)\n", b.addr, b.id)
} else {
Logger.Printf("Connected to broker at %s (unregistered)\n", b.addr)
}
go withRecover(b.responseReceiver)
but in the debug log it looks like metricbeat is can geather this information:
|2018-02-15T07:25:43.813+0100|DEBUG|[kafka]|kafka/broker.go:348|Try to match broker to: 10.107.176.67:20160|
|---|---|---|---|---|
|2018-02-15T07:25:43.813+0100|DEBUG|[kafka]|kafka/broker.go:109|found matching broker 10.107.176.67:20160 with id 2|
|2018-02-15T07:25:43.813+0100|DEBUG|[kafka]|kafka/broker.go:348|Try to match broker to: 10.107.176.70:20160|
|2018-02-15T07:25:43.813+0100|DEBUG|[kafka]|kafka/broker.go:109|found matching broker 10.107.176.70:20160 with id 5|
|2018-02-15T07:25:43.813+0100|DEBUG|[kafka]|kafka/broker.go:348|Try to match broker to: 10.107.176.68:20160|
|2018-02-15T07:25:43.813+0100|DEBUG|[kafka]|kafka/broker.go:109|found matching broker 10.107.176.68:20160 with id 3|
|2018-02-15T07:25:43.817+0100|DEBUG|[kafka]|kafka/broker.go:348|Try to match broker to: 10.107.176.66:20160|
|2018-02-15T07:25:43.817+0100|DEBUG|[kafka]|kafka/broker.go:348|Try to match broker to: 10.107.176.69:20160|
|2018-02-15T07:25:43.817+0100|DEBUG|[kafka]|kafka/broker.go:109|found matching broker 10.107.176.69:20160 with id 4|
|2018-02-15T07:25:43.817+0100|DEBUG|[kafka]|kafka/broker.go:109|found matching broker 10.107.176.66:20160 with id 1|
This indicates either the topics list is empty or an error occured when reading the topics list from the broker. In case of an error when fetching the topics, you will have an 'error' event in elasticsearch.
Which kafka version are you testing with? How many topics do you have in your cluster and how are the topics partitions distributed between brokers?
oh, nice hint. The Topic list is indeet empty. I testet my changes on an empty cluster to prevent several rolling restarts of it. I create some topics an test it again. I come back if i had the chance to test this.
The IP and ID are correct.
Sometimes (after a restart) i have some Events with the following Content:
yes, your point was very valid. The problem exists between keyboard an chair. I tested my changes with an empty cluster. After apply my changes to the production cluster i get metrics again.
Thanks for your support. And sorry that i wasted your time.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.