# Monitor kafka lag

**URL:** <https://discuss.elastic.co/t/monitor-kafka-lag/124106>\
**Category:** Beats\
**Tags:** metricbeat\
**Created:** [March 15, 2018, 1:45pm UTC](https://discuss.elastic.co/t/monitor-kafka-lag/124106 "2018-03-15T13:45:23Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Richard\_Neely](https://avatars.discourse-cdn.com/v4/letter/r/90db22/32.png) [@Richard\_Neely](https://discuss.elastic.co/u/Richard_Neely)\
**Post date:** [March 15, 2018, 1:45pm UTC](https://discuss.elastic.co/t/monitor-kafka-lag/124106/1 "2018-03-15T13:45:23Z")

</div>

I'm looking to calculate lag for kafka which seems like it is kafka.partition.offset.newest - kafka.consumergroup.offset. Those fields are in different documents though across the consumergroup and partition metricset. Is there a way to join those or how would you write the query?

---

<div class="post-metadata">

**Author:** ![Richard\_Neely](https://avatars.discourse-cdn.com/v4/letter/r/90db22/32.png) [@Richard\_Neely](https://discuss.elastic.co/u/Richard_Neely)\
**Post date:** [March 16, 2018, 1:15pm UTC](https://discuss.elastic.co/t/monitor-kafka-lag/124106/2 "2018-03-16T13:15:13Z")

</div>

If we could have an identical field to group with on each document for topic name I think that would work.

---

<div class="post-metadata">

**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\
**Post date:** [March 21, 2018, 6:57am UTC](https://discuss.elastic.co/t/monitor-kafka-lag/124106/3 "2018-03-21T06:57:39Z")

</div>

Hi Richard

The way the Kafka module is working at the moment is not optimal. We plan to mostly redo it based on the feedback we got so far. There are several issues open in the Github repo. It would be great if you could comment with the above for example here: [https://github.com/elastic/beats/issues/3005](https://github.com/elastic/beats/issues/3005)

---

<div class="post-metadata">

**Author:** ![Richard\_Neely](https://avatars.discourse-cdn.com/v4/letter/r/90db22/32.png) [@Richard\_Neely](https://discuss.elastic.co/u/Richard_Neely)\
**Post date:** [March 22, 2018, 6:06pm UTC](https://discuss.elastic.co/t/monitor-kafka-lag/124106/4 "2018-03-22T18:06:58Z")

</div>

I've added it.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 19, 2018, 6:07pm UTC](https://discuss.elastic.co/t/monitor-kafka-lag/124106/5 "2018-04-19T18:07:08Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.