Hello - a quick one hopefully!
Occasionally we experience issues when end users run heavy aggregations - the parent circuit breaker is tripped and this causes issues for other clients which may be performing lighter unrelated queries at the time .
In order to stop the parent breaker being tripped, I'd like to configure circuit breakers at the individual request level to ensure that the offending queries are shut down before they get a chance to cause issues cluster wide .
I have a question about the requests circuit breaker (indices.breaker.request , default value 60%) . The heap usage in the cluster varies from 40%-70% per node approx, there is generally plenty of headroom . I set the indices.breaker.request level to 10% in the hope that this would be sufficient to kill a query before it requested enough memory to trip the parent breaker . I ran some tests and found that I could still trip the parent breaker when running nested aggregations - I needed to lower the request breaker to 5% before it reached a point where the request breaker kicked in before the parent breaker.
So my question is - assuming we have sub aggregations , does each sub aggregation count as an individual request when the requests breaker evaluates memory demand, or is the overall memory demand from the sum of the sub aggregations taken into account? For example, I've seen users create queries with 9 levels of aggregations, so if each of these is evaluated as a single request , it may still be possible to hit the parent breaker threshold even with a modestly sized request breaker threshold.
Cheers