# Kafka metricbeat module -- correct way to access sarama metrics **URL:** \ **Category:** Beats\ **Tags:** metricbeat\ **Created:** [April 20, 2018, 2:22pm UTC](https://discuss.elastic.co/t/kafka-metricbeat-module-correct-way-to-access-sarama-metrics/128893 "2018-04-20T14:22:55Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Filipe\_Oliveira](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/filipe_oliveira/32/30263_2.png) [@Filipe\_Oliveira](https://discuss.elastic.co/u/Filipe_Oliveira)\ **Post date:** [April 20, 2018, 2:22pm UTC](https://discuss.elastic.co/t/kafka-metricbeat-module-correct-way-to-access-sarama-metrics/128893/1 "2018-04-20T14:22:56Z")
Hi there, I wanted to confirm the manner we access metrics via the sarama library and go-metrics. Considering as an example the Broker related metrics we should have two types of metrics ( meter and histogram ). Could you please confirm that for: **Accessing meter values** ( example "incoming-byte-rate" ) inside Fetch() i should proceed like: ``` incomingByteRateBroker := metrics.GetOrRegisterMeter( getMetricNameForBroker( "incoming-byte-rate", b.ID() ), b.Cfg.MetricRegistry ).Rate1() ``` **Accessing histogram values** i would define an new function: ``` func getOrRegisterHistogram(name string, r metrics.Registry) metrics.Histogram { return r.GetOrRegister(name, func() metrics.Histogram { return metrics.NewHistogram(metrics.NewExpDecaySample(metricsReservoirSize, metricsAlphaFactor)) }).(metrics.Histogram) } ``` and inside Fetch(): ``` requestSizeAll := getOrRegisterHistogram( "request-size", b.Cfg.MetricRegistry ) minAll := requestSizeAll.Min() meanAll := requestSizeAll.Mean() maxAll := requestSizeAll.Max() percentile75All := requestSizeAll.Percentile( 75 ) percentile90All := requestSizeAll.Percentile( 90 ) percentile95All := requestSizeAll.Percentile( 0.95 ) percentile99All := requestSizeAll.Percentile( 0.99 ) ``` Are this the correct approaches? Kind regards, ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [April 23, 2018, 3:52pm UTC](https://discuss.elastic.co/t/kafka-metricbeat-module-correct-way-to-access-sarama-metrics/128893/2 "2018-04-23T15:52:15Z")
What's the exact use-case? How is this related to metricbeat? You want metrics from the internal kafka output? libbeat doesn't really make use of go-metrics and tries to wrap it in order to report some selected metrics. ---
**Author:** ![Filipe\_Oliveira](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/filipe_oliveira/32/30263_2.png) [@Filipe\_Oliveira](https://discuss.elastic.co/u/Filipe_Oliveira)\ **Post date:** [April 23, 2018, 11:49pm UTC](https://discuss.elastic.co/t/kafka-metricbeat-module-correct-way-to-access-sarama-metrics/128893/3 "2018-04-23T23:49:08Z")
Hi there, my intent was to extend the kafka module for metricbeat. I would like to confirm the correct way to access the sarama metrics ( used in the kafka module ). Kind regards, ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [April 25, 2018, 10:54am UTC](https://discuss.elastic.co/t/kafka-metricbeat-module-correct-way-to-access-sarama-metrics/128893/4 "2018-04-25T10:54:08Z")
Which kafka module? Given the metrics you proposed, I thought you wanted to collect metrics from the kafka output in metricbeat. The kafka module in metricbeat only uses some of the low-level API in order to query kafka for some information (the broker type from samara only). A module in metricbeat is supposed to query an external service. The kafka module in metricbeat is not supposed to collect internal metrics. Especially, with the metricbeat module not connecting to the full cluster, but one host only. ---
**Author:** ![Filipe\_Oliveira](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/filipe_oliveira/32/30263_2.png) [@Filipe\_Oliveira](https://discuss.elastic.co/u/Filipe_Oliveira)\ **Post date:** [April 25, 2018, 3:27pm UTC](https://discuss.elastic.co/t/kafka-metricbeat-module-correct-way-to-access-sarama-metrics/128893/5 "2018-04-25T15:27:25Z")
Hi there, so what should be the correct approach to collect for example the "request-size" histogram metric from the sarama broker? kind regards, Filipe ---
**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\ **Post date:** [April 25, 2018, 5:33pm UTC](https://discuss.elastic.co/t/kafka-metricbeat-module-correct-way-to-access-sarama-metrics/128893/6 "2018-04-25T17:33:03Z")
TBH I don't really understand what exactly you want to do. Without really understanding what you intend to do I have a hard time to make any recommendations. But let me try. The metricbeat kafka module does not use any of the sarama metrics. Can you add a link to the code you want to insert your custom metrics? Do I understand you right that you want to add your own metrics to this [kafka module](https://github.com/elastic/beats/tree/master/metricbeat/module/kafka) in metricbeat? How/when do you want to report these metrics? You plan to add the internal metrics to the `partition` and `consumergroup` metricset? Metricbeat polls kafka for some stats, depending on the configured period. Do you plan to collect metrics for every 'run' or do you plan to collect these metrics for the lifetime of metrictbeat? In this case, which event do you want to attach these information? The kafka protocol kind of batches up requests for multiple topics/partitions into one request, still metricbeat will report multiple events. Metricbeat [wraps the broker](https://github.com/elastic/beats/blob/master/metricbeat/module/kafka/broker.go#L19). The `NewBroker` function creates a sarama [config object](https://github.com/elastic/beats/blob/master/metricbeat/module/kafka/broker.go#L55). One normally passes a metrics registry to the config object. The config object is reused on `Connect`. The broker is closed after every run. Depending on use-case one could: 1. create a shared registry and reuse it for all active modules/metricsets 2. create a a local registry per broker, but keep it for all runs 3. pass a registry to `Connect`, so to have a new registry on every run. If you use your own custom metrics registry, it will not be processed by metricbeat. You are responsible for adding the processing at the end of a run. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 23, 2018, 5:33pm UTC](https://discuss.elastic.co/t/kafka-metricbeat-module-correct-way-to-access-sarama-metrics/128893/7 "2018-05-23T17:33:05Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.