Httpprof debug/metrics not working anymore for kafka

[ariel.s (Eunseon Song)]

Recently I upgraded filebeat version, and got a problem for monitoring.

v5.0.2
I was using httpprof /debug/metrics to check kafka output rate in one minute for monitoring.
(it is supplied from Sarama kafka lib. )

v5.3.0
/debug/metrics is merged to /debug/vars, but only few metrics.
(libbeat.output.kafka.bytes_write, libbeat.output.kafka.bytes_read, no 'rate' on name, but actually it is rate for whole living time of filebeat)
there is no one-minute rate field.

actually, I just need metrics(ex, Non-zero metrics in the last 30s: ~) on logging file,
but httpprof /debug/vars don't have anything about that.
I think one-minute rate is most important meter it can helps to be noticed various problems from source to target. (application, filebeat, (in this case) kafka)

Now, I start to modify my monitoring script to make one-minute rate(compare "published_events" from previous response and current one...

I hope to be improved this

Or if i missed something, plz let me know how to get one-minute publish rate.
Thx !

[steffens (Steffen Siering)]

the sarama library creating an HTTP handler in /debug/metrics is kind of bad practice. Libraries should not silently register/open endpoints on behalf of the application. The endpoint was not merged into debug/vars, but removed (filtered out), also protecting beats from other libraries potentially installing their own global HTTP handler.

The way metrics are collected and reported by beats and sarama library did differ quite a lot. Beats prefer to report absolute values, no deltas. In ES/Kibana one can use the derivate of the absolute value to compute the rate.

Note: httpprof is no feature you should rely on (it even exposes the golang profiler). Currently it's more like some internal/private developers tools. We're in the process of adding standardised metrics capabilities to beats. This PR even adds an official /stats endpoint to the development branch for future use.

Which metrics is it you're exactly looking for? We're more interested to report similar/comparable metrics for all outputs in beats. As we don't have all of those metrics kafka can potentially report in the other outputs, we did expose only a very small/narrow subset for now.

[ariel.s (Eunseon Song)]

thx to reply.

I'm just interested in published event rate in short time(1 minute) metric.
it is common for all plugins.

I think it's good to supply filebeat logging.metrics (metrics on filebeat log files).

2017-04-28T15:31:59+09:00 INFO Non-zero metrics in the last 30s: publish.events=1413 registrar.states.update=1413 libbeat.kafka.published_and_acked_events=39 libbeat.publisher.published_events=39 registrar.writes=17 libbeat.kafka.call_count.PublishEvents=11

when request like "/metrics", just return last logged one.
It is more useful than tail & parse filebeat logging files.

[steffens (Steffen Siering)]

the logging.metrics are the diff indeed. But you can still compute the diff when collecting or use the derivative in kibana/ES to get exactly the same values.