Bdeus,
I wonder if the consumergroup metrics dont work if there are multiple kafka nodes. Glad to hear that someone else is having the same issue.
Steffen,
Any chance this is a bug that come into play when multiple kafka nodes are being used?
Ok I have confirmed that this works on my local machine with SSL turned on. My test setup is as follows:
-kafka single node with SSL turned on
-Logstash (Same configuration as posted by @steffens ) but with SSL configurations
-metric beats with kafka module with appropriate SSL configs
Result: I am able to get events for both partition and consumergroup
Conclusion: Since this works on my local box and does not in our cluster setup I conclude that kafka module for metric beats does not work for the consumergroup metricset in a clustered kafka configuration. Shall i submit a bug report for this?
1 Like
Yeah, please open an issue on github.
Doing some testing myself and having had a look at the code I think I have an idea what happening. The thing in kafka is a partitions, consumer groups and also consumer group offsets are all handled by different brokers all the time. The problem is, the code neglects the consumer group offsets being handled by another broker, then the consumer group itself.
thankyou @steffens !
I have opened a ticket:
Thanks for your patience and all the testing. I will have to reconsider some decisions/structuring on the kafka module. Eventually we will get the required fields for lag computations + lag itself right from metricbeat, without having to apply to much magic in Kibana.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.