Thanks! This last screengrab you posted is most helpful:
Based on the list of requests you posted, it looks like the 429 is the first error to come back. You haven't posted the response body of the subsequent 500s, but I'd expect those would be a result of the circuit breaking exception you're getting with the first 429.
There are lots of places you can read up on how the circuit breaker works in Elasticsearch, including the docs on circuit breaker settings, this introductory blog post from when the circuit breaker was improved in 7.0, this discuss thread on the topic, or this detailed explanation of how to interpret the error message.
To summarize, the issue is that the request for your visualization is causing the Elasticsearch node to exceed its memory limit as configured by the circuit breaker. Note that this doesn't necessarily suggest an issue with an individual visualization, but it's definitely a possibility. More on that below.
In most cases circuit breaking exceptions need be resolved by doing one of the following:
- Editing visualizations (and/or dashboards) to optimize the queries sent to Elasticsearch. This can be achieved through a careful use of filtering, limiting number of buckets in aggregations, etc.
- Make sure you have a shard sizing strategy to avoid oversharding. The amount of heap memory you have on a node is proportional to the number of shards that node can hold, so if you have too many shards on a node, you'll more quickly hit memory limits and other issues.
- If the above don't work, increase Elasticsearch heap size as explained in the docs (and if necessary also increase node size). It's important that you understand the tradeoffs explained in the (linked) docs if you are going to do this.
You can get more info on your nodes, including circuit breaker settings, using _nodes/stats.
A few other thoughts that come to mind:
- While this might have to do with the complexity your specific visualization, it could also be an issue with the size of your dashboard in general, and this visualization happens to be the one that's triggering the breaker. Do you still get this error when viewing the visualization individually (outside of a dashboard?), or when embedding this same visualization on a smaller dashboard? Or is this error always reproducible with this specific vis?
- Sometimes adjusting the
search.max_bucketscan help with issues like this, either by allowing more buckets (if you know what you're doing), or by allowing less so that you get a more helpful error message. What's unusual here though is that typically you should get a "max buckets exceeded" error instead of a circuit breaking exception if your problem is related to buckets. Might be worth trying as a test though.