I noticed that the Consumer Lag calculation in the [Metrics Kafka] Overview dashboard (8.16.0) provided by the Kafka Integration might be incorrect.
I also have a Kafka UI set up for comparison, and the consumer lag values displayed on the dashboard are significantly lower than those shown in the Kafka UI for each topic.
Upon investigation, I found that the logic behind the lens groups data by topic and then takes the maximum value. However, the consumer lag metrics collected by the Elastic Agent are actually recorded per topic and partition.
Therefore, the correct consumer lag for a topic should be the sum of the consumer lag values across all partitions at the time of collection, rather than simply using the maximum value.
It seems that this logic cannot be implemented using Lens. I attempted to recreate the calculation using TSVB, but I'm unsure whether my approach is accurate or if there's a better way to achieve the correct calculation.
Any suggestions or guidance would be greatly appreciated. Thanks.
-
Default lens of consumer lag in the dashboard.
-
The metric of consumer lag
-
Recreating with TSVB
