Extend Kafka Metricbeat module to add mbean missing

[stevizik (Stef)]

Hi community,

i am using Kafka metricbeat module to get metrics from a Kafka cluster. Furthermore i have enabled Jolokia (JMX - HTTP Connector) to fetch metrics also from broker, consumer and producer.

The solution works as expected but i am missing some critical metrics e.g. kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions

In manifest.yml file (https://github.com/elastic/beats/blob/master/metricbeat/module/kafka/broker/manifest.yml) i can see that there is a way to add more mbean and attach them to fields but i don't know if that is the correct place to look and how i can move on and make the module working with updated metricset.

Thanks!

[jsoriano (Jaime Soriano)]

Hey @stevizik,

This manifest.yml is not intended to be modified (but you can do it You will find it in your installation of Metricbeat, only problem is that it will be overwritten on upgrades).

One thing you can try is to override the jmx.mappings settings from your kafka module configuration. For that, copy these settings from the manifest to the config and there modify or add the metrics you need.

It could be for example something like this in your configuration (not tested):

- module: kafka
  metricsets:
    - broker
  period: 10s
  hosts: ["localhost:8779"]
  jmx.mappings:
        - mbean: 'kafka.network:type=RequestChannel,name=RequestQueueSize'
          attributes:
          - attr: Value
            field: request.channel.queue.size
        ...
        - mbean: 'kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions'
          attributes:
          - attr: Value
            field: replication.under_replicated_partitions

Longer term, it'd be interesting if you could open a pull request in Beats to add this metric to this manifest in the Kafka module, or open an issue requesting to add the fields you are missing