I'm trying to use the Kafka consumergroup metricset to report global lag per consumer group. For simplicity's sake let's say that 1 consumergroup == one topic.
The problem is the lag is reported per partition so if I want a global view of all partitions I need to sum
kafka.consumergroup.consumer_lag and group by
Since I know my metricbeat is configured to send metrics every 30 seconds I can get an accurate number by changing Interval from 'auto' to '30s'. This however, prevents me from displaying data for a long period of time as I reach the max number of buckets. Setting it back to auto lets me get longer periods of time but is inaccurate as several 30s intervals are grouped and summed together.
Can anyone help me with this?