I'd somehow got the idea that Metricbeat 6.5 might enable the monitoring of Kafka consumer lag, but on trying it I see that the latest offset is still only in the partition event and the consumer group offset is still only in the consumergroup record, so this still can't be done?
Hi @TimWard,
In 6.5 we added some fields that help on corelation of events between different Kafka metricsets, this helps on calculating the consumer lag on query/visualization time.
We also added a dashboard that includes a visualization for that, see #8457.
It'd be great if you could try the dashboard and give us feedback 
Thanks for your interest!
Thanks. What do I actually have to do to "try this dashboard"? - I'm not aware of any way of correlating fields between different sets of documents.
(I'm currently calculating lag by parsing the output from kafka-consumer-groups.sh which is horrible for pretty well every reason imaginable.)
Just install the dashboard in kibana and try it with your data 
If you are not using any of the dashboards provided by metricbeat yet, take a look to the setup guide. Even if you don't use them directly they can be used as examples or as a starting point for your own custom dashboards.
If you are using kafka in production it'd be helpful to know if the info provided in the dashboard makes sense when compared to other tools like the scripts provided by kafka.
Ta. I'll give that a try next time I need to upgrade what I'm already getting by parsing the script output, which is good enough for now.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.