# Kafka offset metrics **URL:** \ **Category:** Beats\ **Tags:** metricbeat\ **Created:** [June 24, 2017, 11:24pm UTC](https://discuss.elastic.co/t/kafka-offset-metrics/90731 "2017-06-24T23:24:49Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![John\_06](https://avatars.discourse-cdn.com/v4/letter/j/7c8e57/32.png) [@John\_06](https://discuss.elastic.co/u/John_06)\ **Post date:** [June 24, 2017, 11:24pm UTC](https://discuss.elastic.co/t/kafka-offset-metrics/90731/1 "2017-06-24T23:24:49Z")
I am curious if metrics `kafka.partition.offset.oldest` and `kafka.partition.offset.newest` are the same as `CURRENT-OFFSET` and `LOG_END_OFFSET` respectively? ![](https://us1.discourse-cdn.com/elastic/original/3X/f/e/feae077a8104e675d948b7bba0d3519ef5b90898.png) From console both `CURRENT-OFFSET` and `LOG_END_OFFSET` shows the same value but `kafka.partition.offset.oldest` and `kafka.partition.offset.newest` are different: ![](https://us1.discourse-cdn.com/elastic/original/3X/f/1/f14bccf1a4f3c3227ad29c9d928940b9fe7990a1.png) Is it a bug? ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [June 26, 2017, 11:45am UTC](https://discuss.elastic.co/t/kafka-offset-metrics/90731/2 "2017-06-26T11:45:20Z")
seems like you're mixing conusmer and partition metrics here. The `CURRENT-OFFSET` is the last offset processed by a consumer and `LOG_END_OFFSET`, the last event offset written be a consumer. Kafka is not a real queue in a sense of, consumer once and data is gone. It's storing all data on disk. Every event stored by kafka gets an offset (which is basically an ID, as offset is increased by 1 for every event). When storing events, kafka splits topics into partitions and partitions into segments. Given the retention policy (default by time only) and segment size configurations, kafka will mark old segments as 'deleted' (they are cleaned much later by some GC worker). All of this must be taken into account when sizing kafka (always monitor disk usage. Every segment holds a range of events -\> each segment has a `start` and an `end` offset =\> `# of events in segment = end -start`. Given you have many segments for a partition, the `oldest` offset is the oldest segment it's start offset and newest offset is the most recent/active segment its end offset. That is `# of event stored on disk = kafka.partition.offset.oldest - kafka.partition.offset.newest`. ---
**Author:** ![John\_06](https://avatars.discourse-cdn.com/v4/letter/j/7c8e57/32.png) [@John\_06](https://discuss.elastic.co/u/John_06)\ **Post date:** [June 26, 2017, 9:08pm UTC](https://discuss.elastic.co/t/kafka-offset-metrics/90731/3 "2017-06-26T21:08:31Z")
Hi Steffen, thank you for your explanation. I am wondering what are the proper metrics to calculate a consumer lag then. ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [June 27, 2017, 8:37am UTC](https://discuss.elastic.co/t/kafka-offset-metrics/90731/4 "2017-06-27T08:37:30Z")
`CONSUMER LAG = LOG_END_OFFSET - CURRENT-OFFSET`. The lag is to be computed per conusmer group (LEFT JOIN on topic, partition, consumer group). As each partition acts as a queue itself, the lag is to be computed per partition. Having lag per partition, one can compute total consumer lag and max consumer lag + compare lag for being hopefully about evenly distributed between partitions for one conusmer group. In theory the consumergroup metricset can be used to get the missing stats in metricbeat. But unfortunately this feature (still in beta btw.) [contains a bug, not properly collecting the data](https://github.com/elastic/beats/issues/4285). ---
**Author:** ![John\_06](https://avatars.discourse-cdn.com/v4/letter/j/7c8e57/32.png) [@John\_06](https://discuss.elastic.co/u/John_06)\ **Post date:** [June 27, 2017, 5:50pm UTC](https://discuss.elastic.co/t/kafka-offset-metrics/90731/5 "2017-06-27T17:50:09Z")
Thanks. Btw, I can see both metrics `kafka.partition.offset.oldest` and `kafka.partition.offset.newest` have the same value now after 2 days. And the same chart looks very different today (first screen-shot is from June 24th, around 16:00) ![](https://us1.discourse-cdn.com/elastic/original/3X/a/7/a7ad56488f9fdd456e21cf40c4ffd8610e2ee9b0.png) ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [June 28, 2017, 10:10am UTC](https://discuss.elastic.co/t/kafka-offset-metrics/90731/6 "2017-06-28T10:10:24Z")
The kafka distribution includes some command line tools you can use to query your kafka cluster for partitions and offsets. These are the offets as reported by kafka. To me it looks like data/segments have finally been deleted due to time-based retention policy? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 26, 2017, 10:10am UTC](https://discuss.elastic.co/t/kafka-offset-metrics/90731/7 "2017-07-26T10:10:42Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.