Consumer Groups not reporting offset topics

Elastic Stack Beats
1

Hi All,

Got a problem with metricbeat kafka consumer groups offset for topics/partition.
After updating Kafka from 0.10.0.1 to 0.10.2.1, stoped reporting offset.

Bellow part of debug log related to kafka module:

2017/08/01 14:30:50.572196 wrapper.go:128: DBG  Starting Wrapper[name=kafka, len(metricSetWrappers)=1]
2017/08/01 14:30:50.572378 wrapper.go:149: DBG  Stopped Wrapper[name=kafka, len(metricSetWrappers)=1]
2017/08/01 14:30:50.572616 reload.go:200: DBG  New runner started: 11180739783407736849
2017/08/01 14:30:50.574135 wrapper.go:186: DBG  Starting metricSetWrapper[module=kafka, name=consumergroup, host=brokerip:9092]
2017/08/01 14:30:50.576531 log.go:12: WARN Connected to broker at brokerip:9092 (unregistered)
2017/08/01 14:30:50.588645 broker.go:340: DBG  Try to match broker to: brokerip:9092
2017/08/01 14:30:50.588819 broker.go:108: DBG  found matching broker brokerip:9092 with id 0
2017/08/01 14:30:50.593368 query.go:38: DBG  known consumer groups: %!(EXTRA []string=[consumergroup, consumergroup2, consumergroup3])
2017/08/01 14:30:50.634408 log.go:12: WARN Closed connection to broker brokerip:9092

After diging in source code of metric found that https://github.com/elastic/beats/blob/master/metricbeat/module/kafka/broker.go#L233-L239 return nil and that why offsets is empty.
Opened ticket for sarama https://github.com/Shopify/sarama/issues/921 and they said there's need specify topic and partition in order to get offset.

Is there's any possible to overcome issue?

2

how many brokers do you have in your kafka cluster.

There is a bug in the metricset collecting information from clusters having more then one broker, as part of the meta-data is hold/managed by another broker.

3

he've clusters with different amount of brokers. (1,4,6,9) nodes
And stoped reporting on > 4 nodes cluster exactly after update, on old 4 cluster we can still get from broker metadata for consumer groups.

And i tried to figure out about getting metadata with consumer and seams there's an API for getting metadata (topics for consumer groups)

4

Thanks for the details. For the cluster with brokers > 1, it sounds to be exactly this ticket. In the original discuss topic we've been looking at the issue and found the problem to be the distribution of state between brokers and metricbeat not asking remote brokers for meta data if it really needs to do so. It's quite a long discussion, but still an interesting read. To my experience with the API, you're quite lucky it did work for you with 0.10.1, as normally it's more by chance if the number of brokers is > 1.

This brings me to the single broker cluster. Which version are you using with the single broker cluster. Did you test with single broker cluster, version 0.10.2 and still found it failing?

Please note, the feature is in beta right now and we are aware of some issues with it. Unfortunately we haven't had time to work on the metricset yet.

Thanks for reporting your experience and versions being used. This discussion adds quite some valuable input for things to look out for potential problems and tests when I finally find time to improve the metricset.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.