Sum bucket of filters no longer works in kibana 7.11

[Matthew_Field]

Since upgrading to kibana 7.11 my sibling sum bucket aggregations no longer work.

Following upgrade the visualisations that were previously working were all blank.

I am trying now to recreate the same visualisations as i had before using sum bucket with filters, but the results return zero. However if i filter the visualisation by the same filter, then they no longer return zero but return the correct value.

Thanks in advance for any help.

[wylie (Wylie Conlon)]

You didn't provide any details about how you upgraded, but users sometimes don't match up the index pattern + visualization IDs when recreating visualizations in Kibana. But to focus on the question you asked about the sum being zero: What does the Inspect tab show is being requested?

[Matthew_Field]

Thanks for your interest Wylie,

Here is the request:

{
  "aggs": {
    "2": {
      "terms": {
        "field": "exchange",
        "order": {
          "_count": "desc"
        },
        "size": 5
      },
      "aggs": {
        "3": {
          "sum_bucket": {
            "buckets_path": "3-bucket>_count"
          }
        },
        "3-bucket": {
          "filters": {
            "filters": {
              "event_type:\"TRADE\"": {
                "bool": {
                  "must": [],
                  "filter": [
                    {
                      "bool": {
                        "should": [
                          {
                            "match_phrase": {
                              "event_type": "TRADE"
                            }
                          }
                        ],
                        "minimum_should_match": 1
                      }
                    }
                  ],
                  "should": [],
                  "must_not": []
                }
              }
            }
          }
        }
      }
    }
  },
  "size": 0,
  "fields": [
    {
      "field": "@timestamp",
      "format": "date_time"
    }
  ],
  "script_fields": {},
  "stored_fields": [
    "*"
  ],
  "_source": {
    "excludes": []
  },
  "query": {
    "bool": {
      "must": [],
      "filter": [
        {
          "match_all": {}
        },
        {
          "range": {
            "@timestamp": {
              "gte": "2021-02-23T17:34:45.678Z",
              "lte": "2021-02-23T17:49:45.678Z",
              "format": "strict_date_optional_time"
            }
          }
        }
      ],
      "should": [],
      "must_not": []
    }
  }
}

and the response

{
  "took": 606,
  "timed_out": false,
  "_shards": {
    "total": 298,
    "successful": 298,
    "skipped": 286,
    "failed": 0
  },
  "hits": {
    "total": 13830873,
    "max_score": null,
    "hits": []
  },
  "aggregations": {
    "2": {
      "doc_count_error_upper_bound": 97798,
      "sum_other_doc_count": 5683926,
      "buckets": [
        {
          "3": {
            "value": 0
          },
          "key": "binancefut",
          "doc_count": 1997919,
          "3-bucket": {
            "buckets": {
              "event_type:\"TRADE\"": {
                "doc_count": 0
              }
            }
          }
        },
        {
          "3": {
            "value": 0
          },
          "key": "binance",
          "doc_count": 1987809,
          "3-bucket": {
            "buckets": {
              "event_type:\"TRADE\"": {
                "doc_count": 0
              }
            }
          }
        },
        {
          "3": {
            "value": 0
          },
          "key": "ftx",
          "doc_count": 1634364,
          "3-bucket": {
            "buckets": {
              "event_type:\"TRADE\"": {
                "doc_count": 0
              }
            }
          }
        },
        {
          "3": {
            "value": 0
          },
          "key": "kraken",
          "doc_count": 1308034,
          "3-bucket": {
            "buckets": {
              "event_type:\"TRADE\"": {
                "doc_count": 0
              }
            }
          }
        },
        {
          "3": {
            "value": 0
          },
          "key": "gdax",
          "doc_count": 1218813,
          "3-bucket": {
            "buckets": {
              "event_type:\"TRADE\"": {
                "doc_count": 0
              }
            }
          }
        }
      ]
    }
  }
}

Ah.. now i see that doc_count error... could that be my issue? there are only a relatively small amount of docs there. (less than 100 out of a total of 13 million..)

[wylie (Wylie Conlon)]

It seems like your filters are legitimately matching zero documents per exchange within the time period you were querying. The error_upper_bound is about the potential inaccuracy in the most-frequently-occuring terms. Terms aggregation | Elasticsearch Reference [7.11] | Elastic

What I would suggest doing is making a similar query using Discover. Look for event_type:TRADE and exchange:binancefut within the time period.

[Matthew_Field]

Yes.. the thing is .. what i am trying to do is to compare (small) error rates with big volumes of documents so i want to have both on the same table.. do you know how i could do that without having to do it manually?
ie:

group1 errors total number of docs
group 2 errors total number of docs

[wylie (Wylie Conlon)]

Your chart config is pretty much showing that already