Kafka metricbeat module -- correct way to access sarama metrics

[Filipe_Oliveira]

Hi there,
I wanted to confirm the manner we access metrics via the sarama library and go-metrics.
Considering as an example the Broker related metrics we should have two types of metrics ( meter and histogram ).

Could you please confirm that for:
Accessing meter values
( example "incoming-byte-rate" ) inside Fetch() i should proceed like:

incomingByteRateBroker := metrics.GetOrRegisterMeter( getMetricNameForBroker( "incoming-byte-rate", b.ID() ), b.Cfg.MetricRegistry ).Rate1()

Accessing histogram values
i would define an new function:

func getOrRegisterHistogram(name string, r metrics.Registry) metrics.Histogram {
	return r.GetOrRegister(name, func() metrics.Histogram {
		return metrics.NewHistogram(metrics.NewExpDecaySample(metricsReservoirSize, metricsAlphaFactor))
}).(metrics.Histogram)
}

and inside Fetch():

requestSizeAll := getOrRegisterHistogram( "request-size", b.Cfg.MetricRegistry )

minAll := requestSizeAll.Min()
meanAll := requestSizeAll.Mean()
maxAll := requestSizeAll.Max()

percentile75All := requestSizeAll.Percentile( 75 )
percentile90All	:= requestSizeAll.Percentile( 90 )
percentile95All := requestSizeAll.Percentile( 0.95 )
percentile99All := requestSizeAll.Percentile( 0.99 )

Are this the correct approaches?
Kind regards,

[steffens (Steffen Siering)]

What's the exact use-case? How is this related to metricbeat? You want metrics from the internal kafka output?

libbeat doesn't really make use of go-metrics and tries to wrap it in order to report some selected metrics.

[Filipe_Oliveira]

Hi there,
my intent was to extend the kafka module for metricbeat. I would like to confirm the correct way to access the sarama metrics ( used in the kafka module ).

Kind regards,

[steffens (Steffen Siering)]

Which kafka module?

Given the metrics you proposed, I thought you wanted to collect metrics from the kafka output in metricbeat.

The kafka module in metricbeat only uses some of the low-level API in order to query kafka for some information (the broker type from samara only). A module in metricbeat is supposed to query an external service. The kafka module in metricbeat is not supposed to collect internal metrics. Especially, with the metricbeat module not connecting to the full cluster, but one host only.

[Filipe_Oliveira]

Hi there,
so what should be the correct approach to collect for example the "request-size" histogram metric from the sarama broker?
kind regards,
Filipe

[steffens (Steffen Siering)]

TBH I don't really understand what exactly you want to do. Without really understanding what you intend to do I have a hard time to make any recommendations. But let me try.

The metricbeat kafka module does not use any of the sarama metrics.

Can you add a link to the code you want to insert your custom metrics?

Do I understand you right that you want to add your own metrics to this kafka module in metricbeat?

How/when do you want to report these metrics?

You plan to add the internal metrics to the partition and consumergroup metricset?

Metricbeat polls kafka for some stats, depending on the configured period. Do you plan to collect metrics for every 'run' or do you plan to collect these metrics for the lifetime of metrictbeat? In this case, which event do you want to attach these information? The kafka protocol kind of batches up requests for multiple topics/partitions into one request, still metricbeat will report multiple events.

Metricbeat wraps the broker. The NewBroker function creates a sarama config object. One normally passes a metrics registry to the config object. The config object is reused on Connect. The broker is closed after every run. Depending on use-case one could:

1. create a shared registry and reuse it for all active modules/metricsets
2. create a a local registry per broker, but keep it for all runs
3. pass a registry to Connect, so to have a new registry on every run.

If you use your own custom metrics registry, it will not be processed by metricbeat. You are responsible for adding the processing at the end of a run.