Trouble Visualizing API Response Times in Percentiles with Prometheus Data in Elasticsearch

Elastic Stack Kibana
,
1

  • Issue: I am trying to visualize API response times in percentiles using Prometheus histogram data stored in Elasticsearch. Despite sending data from Prometheus to Elasticsearch, I am encountering difficulties in creating meaningful visualizations.
    Kibana Version: 8.4.3

  • Approach:

  • I am using Prometheus to collect histogram data (api_response_time_bucket).

  • Data is then exported to Elasticsearch for storage and visualization in Kibana.

  • Response times are stored in cumulative buckets using the le (less than or equal) operator in Prometheus.

  • Challenge: I am unsure about the correct approach to extract and visualize percentiles (like 90th percentile) from histogram data stored in Elasticsearch.

  • Desired Outcome: My goal is to create Kibana visualizations that show percentile response times over time, segmented by API endpoints.

  • Could anyone provide guidance on how to properly extract percentile metrics from Prometheus histogram data stored in Elasticsearch, particularly when using cumulative buckets with the le operator?

  • Any examples, suggestions or alternative approach on setting up Kibana visualizations for percentile API response times would be greatly appreciated.

api_response_time_bucket{endpoint="/ep1", le="10"} 5
api_response_time_bucket{endpoint="/ep1", le="25"} 5
api_response_time_bucket{endpoint="/ep1", le="50"} 15
api_response_time_bucket{endpoint="/ep1", le="100"} 15
api_response_time_bucket{endpoint="/ep1", le="1000"} 15
api_response_time_bucket{endpoint="/ep1", le="+Inf"} 15
2

Added dashboard, visualisation

3

Hi @KKarjee

can you share the mapping of the index?

4

Hi @Marco_Liberati, here are the relevant fields of the mapping. The actual size is too large to include

{
    "metrics-xxx": {
      "mappings": {
        "dynamic": "true",
        "_meta": {
          "date_detection": true,
          "properties": {
            "@timestamp": {
              "format": "strict_date_optional_time",
              "type": "date"
            },
            "prometheus": {
              "properties": {
                "labels": {
                  "properties": {
                    "endpoint": {
                        "type": "keyword"
                      },
                      "status": {
                        "type": "keyword"
                      },
                      "le": {
                        "type": "keyword"
                    }
                  }
                },
                "metrics": {
                  "properties": {
                    "api_counter": {
                        "type": "double"
                    },
                    "response_time_bucket": {
                        "type": "double"
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }

sample

{
    "_source": {
        "prometheus": {
            "metrics": {
                "response_time_bucket": 10
            },
            "labels": {
                "endpoint": "/ep1",
                "status": "Success",
                "le": "50"
            }
        }
    }
}

{
    "_source": {
        "prometheus": {
            "metrics": {
                "response_time_bucket": 15
            },
            "labels": {
                "endpoint": "/ep1",
                "status": "Success",
                "le": "250"
            }
        }
    }
}

{
    "_source": {
        "prometheus": {
            "metrics": {
                "response_time_bucket": 15
            },
            "labels": {
                "endpoint": "/ep1",
                "status": "Success",
                "le": "5000"
            }
        }
    }
}
5

If I understood correctly you want to use the le field to programmatically set the percentile ranking of a request.
Did I get it correctly?

6

That's right.