Too_many_buckets_exception after patching to 7.x

I'm getting this error in our some of our Grafana panels now after updating to 7.4 from 6.8:

type:"too_many_buckets_exception"

reason:"Trying to create too many buckets. Must be less than or equal to: [10000] but was [10001]. This limit can be set by changing the [search.max_buckets] cluster level setting."

Elastic ref says:

The maximum number of buckets allowed in a single response is limited by a dynamic cluster setting namedsearch.max_buckets. It defaults to 10,000, requests that try to return more than the limit will fail with an exception

Seeing this issue also after updating to 7.x, mentioning: 'So be careful about making the limit too high'

What might be considered too high, when nodes trips over with OOM or...?

You should decrease time period or increase group interval for charts in order to decrease buckets count. Also Grafana should use composite aggregator or scroll API

time period is hard to decrease as grafana users can select a 'large' time periods to view many data points.

Something most have changed in 7.x vs. previous version as these specific grafana panels have always worked with default search.max_buckets in version 2.x, 5.x and 6.x.

Now I've increased search.max_buckets to > 35000 then panels seems to work again also showing larger time periods.

According to breaking 7.0 changes
https://www.elastic.co/guide/en/elasticsearch/reference/7.0/breaking-changes-7.0.html#search-max-buckets-cluster-setting

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.