Too_many_buckets_exception after patching to 7.x

Elastic Stack Elasticsearch
1

I'm getting this error in our some of our Grafana panels now after updating to 7.4 from 6.8:

type:"too_many_buckets_exception"

reason:"Trying to create too many buckets. Must be less than or equal to: [10000] but was [10001]. This limit can be set by changing the [search.max_buckets] cluster level setting."

Elastic ref says:

The maximum number of buckets allowed in a single response is limited by a dynamic cluster setting namedsearch.max_buckets. It defaults to 10,000, requests that try to return more than the limit will fail with an exception

Seeing this issue also after updating to 7.x, mentioning: 'So be careful about making the limit too high'

What might be considered too high, when nodes trips over with OOM or...?

2

You should decrease time period or increase group interval for charts in order to decrease buckets count. Also Grafana should use composite aggregator or scroll API

3

time period is hard to decrease as grafana users can select a 'large' time periods to view many data points.

Something most have changed in 7.x vs. previous version as these specific grafana panels have always worked with default search.max_buckets in version 2.x, 5.x and 6.x.

Now I've increased search.max_buckets to > 35000 then panels seems to work again also showing larger time periods.

4

According to breaking 7.0 changes
https://www.elastic.co/guide/en/elasticsearch/reference/7.0/breaking-changes-7.0.html#search-max-buckets-cluster-setting