Hello,
Elastic offers a Prometheus Metricbeat (https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-prometheus.html) module that scrapes a Prometheus /metrics endpoint and pushes the metrics to an Elastic Cloud endpoint. This is all working fine.
How is it possible to visualise Prometheus histograms in Kibana? There is a Prometheus scraping example (https://github.com/elastic/examples/tree/master/scraping-prometheus-k8s-with-metricbeat) but this only shows how to visualise a count metric.
Prometheus histograms are already bucketed using a le (less than or equal to) field. Each bucket contains a count of events that are "less than or equal to" that bucket. The last bucket is always +Inf which will always contain all counts (as everything is "less than or equal to" infinity).
Some example documents that arrive from the Prometheus metricbeat:
{
"_index": "metricbeat-7.6.2-2020.05.10-000001",
"_type": "_doc",
"_version": 1,
"_score": null,
"_source": {
"prometheus": {
"labels": {
"le": "0.064"
},
"metrics": {
"latency1_seconds_bucket": 17
}
}
}
}
{
"_index": "metricbeat-7.6.2-2020.05.10-000001",
"_type": "_doc",
"_version": 1,
"_score": null,
"_source": {
"prometheus": {
"labels": {
"le": "0.128"
},
"metrics": {
"latency1_seconds_bucket": 23
}
}
}
}
The above documents mean 17 events took <= 0.064 seconds and 23 events took <= 0.128 seconds. This means 6 events took between 0.064 and 0.128 seconds.
How is it possible to plot something like a percentile of latencies out of this data? I couldn't find any documentation about this in the Metricbeat examples/docs.
Thank you,
Iulian