Hi im setting up an ELK monitoring for my kafka cluster of 5 broker/controller and prometheus.
I have this issue where the active controller will report as per usual with the other brokers however after some time, some data will be missing from kibana/elasticsearch from the active controller. When i search via the prometheus endpoint for that controller(which is what my metric beat is configured to) i can see the metric. I can see it in the metric beat logs in debug mode as well.
Im looking for prometheus.metrics.kafka_controller_kafkacontroller_activecontrollercount which should be 1 when controller is active. However it disappears after an inconsistent amount of time. Some other metrics disappear as well. Anyone has any idea why?