Performance issue with terms sub-aggregations when using global ordinals

[avencar (Andre E)]

I am seeing poor performance when using the GlobalOrdinalsStringTermsAggregator in sub-aggregations, in comparison to the StringTermsAggregation. In particular, when doing a GlobalOrdinalsStringTermsAggregation as a sub-aggregation, the run-time is much longer and often times causes elasticsearch to memory thrash (GC runs very often, and for long periods of time, and eventually can result in a OOM error).

I put together a simple example below to demonstrate the issue, along with results from the elasticsearch query profiler. The difference is entirely attributable to the "collect" phase of the aggregation.

Any idea what could be going on? For now, I can avoid the issue by using "execution_hint" set to "map". But I can't understand why that results in drastically better performance.

Thanks in advance for the help!

Some relevant context on my example:

• elasticsearch version 6.8.4
• query executed across millions of documents
• field1 is a high-cardinality field (millions of unique values)
• field2 is a low cardinality field (hundreds of unique values)

Query:

curl -XGET "http://localhost:9200/data-*/_search?pretty" -H 'Content-Type: application/json' -d'
{
  "size":0,
  "profile": true,
  "aggs": {
    "f1": {
      "terms": {
        "field": "field1.keyword",
        "size": 1000
      },
      "aggs": {
        "s_ordinal": {
          "terms": {
            "field": "field2.keyword",
            "size": 1000
          }
        },
        "s_map": {
          "terms": {
            "field": "field3.keyword",
            "size": 1000,
            "execution_hint": "map"
          }
        }
      }
    }
  }
}'

Relevant result from the profiler:

"aggregations" : [
          {
            "type" : "GlobalOrdinalsStringTermsAggregator",
            "description" : "f1",
            "time_in_nanos" : 57230900779,
            "breakdown" : {
              "reduce" : 0,
              "build_aggregation" : 55499491624,
              "build_aggregation_count" : 1,
              "initialize" : 51224,
              "initialize_count" : 1,
              "reduce_count" : 0,
              "collect" : 1718063838,
              "collect_count" : 13294091
            },
            "children" : [
              {
                "type" : "GlobalOrdinalsStringTermsAggregator",
                "description" : "s_ordinal",
                "time_in_nanos" : 42106237199,
                "breakdown" : {
                  "reduce" : 0,
                  "build_aggregation" : 7187184,
                  "build_aggregation_count" : 1510,
                  "initialize" : 4833,
                  "initialize_count" : 1,
                  "reduce_count" : 0,
                  "collect" : 42086957995,
                  "collect_count" : 12085676
                }
              },
              {
                "type" : "StringTermsAggregator",
                "description" : "s_map",
                "time_in_nanos" : 8283080112,
                "breakdown" : {
                  "reduce" : 0,
                  "build_aggregation" : 3413487,
                  "build_aggregation_count" : 1510,
                  "initialize" : 1557,
                  "initialize_count" : 1,
                  "reduce_count" : 0,
                  "collect" : 8267577881,
                  "collect_count" : 12085676
                }
              }
            ]
          }
        ]