Ordering by percentile not working anymore

Elastic Stack Kibana
1

Hi!
I had a couple of Lens line graphs of some response times set up, ordered by the 95th percentile. In recent days I noticed they don't work anymore, although I didn't change anything (they were last changed in January). It seems like the graph starts with some hostnames that don't have any datapoints in my time range for this particular filtered set of data. Does anyone know a quick way to fix them?
Thanks in advance!

2

Hi @VeraE ,

have you upgraded the stack recently? What version of the stack are you using right now?

As of the Lens configuration, do you have more info to share? I.e. have you configured the percentile metric via formula?

3

Hi @Marco_Liberati!
I don't know exactly if and when we upgraded, we are using Elastic cloud, and now it's on Version 8.3.1.
I configured it with a Quick function. It is really easy, just a 95th percentile of opening time (just double checked, even without filtering), with hostname on the x axis. The first hosts don't have any data in this particular field for the given time period. If I expand the number of hosts to show, at one point I have a graph that looks like what I expect, but with the empty ones in the front, it's not usable. Forgot to mention that this is descending, if I set it to ascending, it also seems to work.

4

Percentiles are not caluclated with 100% precision in Elasticsearch for performance reasons, maybe this is what's going on here and 8.3 changed how it works exactly. Could you copy/paste the request and response from the "Inspect" menu for the config that's not working?

5

Sure, here you go:
Request:

{
  "aggs": {
    "0": {
      "terms": {
        "field": "host.keyword",
        "order": {
          "1.95": "desc"
        },
        "size": 1000
      },
      "aggs": {
        "1": {
          "percentiles": {
            "field": "OpeningTimeInMillis",
            "percents": [
              95
            ]
          }
        }
      }
    }
  },
  "size": 0,
  "fields": [
    {
      "field": "@timestamp",
      "format": "date_time"
    }
  ],
  "script_fields": {},
  "stored_fields": [
    "*"
  ],
  "runtime_mappings": {},
  "_source": {
    "excludes": []
  },
  "query": {
    "bool": {
      "must": [],
      "filter": [
        {
          "range": {
            "@timestamp": {
              "format": "strict_date_optional_time",
              "gte": "2022-07-07T08:21:23.845Z",
              "lte": "2022-07-07T08:36:23.845Z"
            }
          }
        }
      ],
      "should": [],
      "must_not": []
    }
  }
}

Response:

{
  "id": "Fkp4c3pjM0hvUy0tQ25xVWpWOXVpcVEgSFl5YzBoTXpRT1N2OGJ4dzM2RFl0ZzoyOTM4NzI0NzA=",
  "rawResponse": {
    "took": 405,
    "timed_out": false,
    "_shards": {
      "total": 3,
      "successful": 3,
      "skipped": 2,
      "failed": 0
    },
    "hits": {
      "total": 2026,
      "max_score": null,
      "hits": []
    },
    "aggregations": {
      "0": {
        "doc_count_error_upper_bound": 0,
        "sum_other_doc_count": 0,
        "buckets": [
          {
            "1": {
              "values": {
                "95.0": null
              }
            },
            "key": "host2498",
            "doc_count": 1
          },
          {
            "1": {
              "values": {
                "95.0": null
              }
            },
            "key": "host2538",
            "doc_count": 1
          },
          {
            "1": {
              "values": {
                "95.0": null
              }
            },
            "key": "host2733",
            "doc_count": 1
          },
          {
            "1": {
              "values": {
                "95.0": null
              }
            },
            "key": "host3563",
            "doc_count": 1
          },
          {
            "1": {
              "values": {
                "95.0": 14982
              }
            },
            "key": "host8876",
            "doc_count": 4
          },
          {
            "1": {
              "values": {
                "95.0": 6345
              }
            },
            "key": "host5838",
            "doc_count": 6
          },
          {
            "1": {
              "values": {
                "95.0": 5742
              }
            },
            "key": "host3051",
            "doc_count": 2
          },
          {
            "1": {
              "values": {
                "95.0": 5740
              }
            },
            "key": "host2729",
            "doc_count": 10
          },
          {
            "1": {
              "values": {
                "95.0": 938
              }
            },
            "key": "host5573",
            "doc_count": 3
          },
          {
            "1": {
              "values": {
                "95.0": 887
              }
            },
            "key": "host8438",
            "doc_count": 3
          }
        ]
      }
    }
  },
  "isPartial": false,
  "isRunning": false,
  "total": 3,
  "loaded": 3,
  "isRestored": true
}

I trimmed the response a bit, cause it was ~3000 lines (just deleted some in-between values, so it's shorter).
I created it in January, it worked up until last week. The problem clearly seems to be that during ordering the null values are put in front.

6

Hi @flash1293! Is there any progress on this?

7

Hi VeraE, sorry for the late reply. It seems like the data is shipped from Elasticsearch like this - could you open a bug report here? GitHub - elastic/elasticsearch: Free and Open, Distributed, RESTful Search Engine

8

I did just that, thanks!
See: Ordering by percentile not working anymore · Issue #89079 · elastic/elasticsearch · GitHub

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.