# Elasticsearch Request Circuit Breaker
**URL:** https://discuss.elastic.co/t/elasticsearch-request-circuit-breaker/275893
**Category:** Elasticsearch
**Created:** [June 14, 2021, 7:15pm UTC](https://discuss.elastic.co/t/elasticsearch-request-circuit-breaker/275893 "2021-06-14T19:15:17Z")
**Posts on this page:** 14
**Page:** 1
### Author:  [@animageofmine](https://discuss.elastic.co/u/animageofmine)
#### Post date: [June 14, 2021, 7:15pm UTC](https://discuss.elastic.co/t/elasticsearch-request-circuit-breaker/275893/1 "2021-06-14T19:15:17Z")
I am looking at **indices.breaker.request.limit** [circuit breaker](https://www.elastic.co/guide/en/elasticsearch/reference/6.8/circuit-breaker.html#request-circuit-breaker). Based on the description, my understanding is that it would break if an aggregation consumes the heap configured in the breaker. The default is 60% for version 6.8.
We are trying large cardinality terms aggregation. The metrics show that it barely goes beyond a handful of megabytes but the cluster seems to turn red. I am fairly certain it takes more memory for larger cardinality aggregations.
Wondering if I am misunderstanding the circuit breaker, if there's a known bug or I am simply missing something. Your insight would be helpful. Thank you.
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [June 15, 2021, 2:01am UTC](https://discuss.elastic.co/t/elasticsearch-request-circuit-breaker/275893/2 "2021-06-15T02:01:04Z")
> [@animageofmine](#):
>
> The metrics show that it barely goes beyond a handful of megabytes but the cluster seems to turn red
What do the Elasticsearch logs show?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [June 15, 2021, 4:17am UTC](https://discuss.elastic.co/t/elasticsearch-request-circuit-breaker/275893/3 "2021-06-15T04:17:08Z")
I believe at least some circuit breakers estimates the amount of memory required and triggers if this is too large rather that wait until it goes beyond a certain limit.
---
### Author:  [@animageofmine](https://discuss.elastic.co/u/animageofmine)
#### Post date: [June 15, 2021, 11:51pm UTC](https://discuss.elastic.co/t/elasticsearch-request-circuit-breaker/275893/4 "2021-06-15T23:51:34Z")
Log shows Out of heap space. Nothing about circuit breaker though.
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [June 15, 2021, 11:59pm UTC](https://discuss.elastic.co/t/elasticsearch-request-circuit-breaker/275893/5 "2021-06-15T23:59:55Z")
It'd be useful to share the logs.
---
### Author:  [@animageofmine](https://discuss.elastic.co/u/animageofmine)
#### Post date: [June 16, 2021, 10:20pm UTC](https://discuss.elastic.co/t/elasticsearch-request-circuit-breaker/275893/6 "2021-06-16T22:20:46Z")
Circuit breaker
Logs and heap space. node b3 (data node) consumed ~29G heap space (was about 14 and spiked up quickly because of high cardinality request). Eventually it disconnected itself from the cluster (master probably kicked it out after timeout). The uber point is that the circuit breaker numbers are not even close to the heap space consumed by the large cardinality requests. The breaker didn't trip and protect the cluster.
We tried the inflight breaker setting to 1G & 2G.
```auto
[2021-06-16T21:13:07,729][WARN][o.e.m.j.JvmGcMonitorService] [esdata-b3-cluster] [gc][old][537][5] duration [1.6m], collections [1]/[1.6m], total [1.6m]/[3m], memory [27.5gb]->[28.4gb]/[29.7gb], all_pools {[young] [449.4kb]->[712mb]/[1.8gb]}{[survivor] [232.9mb]->[0b]/[232.9mb]}{[old] [27.3gb]->[27.7gb]/[27.7gb]}
[2021-06-16T21:11:26,065][WARN][o.e.m.j.JvmGcMonitorService] [esdata-b3] [gc][old][535][4] duration [1.4m], collections [1]/[1.4m], total [1.4m]/[1.4m], memory [27.2gb]->[26.2gb]/[29.7gb], all_pools {[young] [50.7mb]->[39.5mb]/[1.8gb]}{[survivor] [232.9mb]->[0b]/[232.9mb]}{[old] [27gb]->[26.2gb]/[27.7gb]}
.
.// more logs in between
.
[2021-06-16T21:09:39,557][WARN][o.e.m.j.JvmGcMonitorService] [esdata-b3-cluster] [gc][young][523][30] duration [1.6s], collections [1]/[1.9s], total [1.6s]/[38.6s], memory [14.9gb]->[15.9gb]/[29.7gb], all_pools {[young] [11.7mb]->[1.9mb]/[1.8gb]}{[survivor] [232.9mb]->[232.9mb]/[232.9mb]}{[old] [14.6gb]->[15.7gb]/[27.7gb]}
```
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [June 17, 2021, 7:49am UTC](https://discuss.elastic.co/t/elasticsearch-request-circuit-breaker/275893/7 "2021-06-17T07:49:52Z")
> [@animageofmine](#):
>
> Wondering if I am misunderstanding the circuit breaker, if there's a known bug or I am simply missing something.
I think you're misunderstanding the circuit breaker: it does not account for every memory consumer so will miss things sometimes. Note that the 6.8 series was released over two years ago so you're missing out on two years of further development in this area. Recent versions track memory usage much more accurately.
---
### Author:  [@animageofmine](https://discuss.elastic.co/u/animageofmine)
#### Post date: [June 17, 2021, 6:30pm UTC](https://discuss.elastic.co/t/elasticsearch-request-circuit-breaker/275893/8 "2021-06-17T18:30:21Z")
Thanks. Would it miss the cardinality aggregation? That is the specific aggregation that's tipping over the node (high memory usage).
Also, please let me know if there is a detailed documentation about the circuit breaker I can read (what does it account for and what it does not).
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [June 17, 2021, 7:29pm UTC](https://discuss.elastic.co/t/elasticsearch-request-circuit-breaker/275893/9 "2021-06-17T19:29:05Z")
> [@animageofmine](#):
>
> Would it miss the cardinality aggregation?
I'm not sure, but it sounds from your account that it does.
> [@animageofmine](#):
>
> Also, please let me know if there is a detailed documentation about the circuit breaker I can read (what does it account for and what it does not).
The big change in newer versions is described in this blog post:
> **[Improve Elasticsearch resiliency with the real memory circuit breaker](https://www.elastic.co/blog/improving-node-resiliency-with-the-real-memory-circuit-breaker)**
>
> We're excited to announce a new circuit breaker implementation available in Elasticsearch 7.0.0 that will improve resiliency of single nodes within your cluster.
With that change the parent circuit breaker effectively tracks everything on the heap.
---
### Author:  [@animageofmine](https://discuss.elastic.co/u/animageofmine)
#### Post date: [June 18, 2021, 6:18am UTC](https://discuss.elastic.co/t/elasticsearch-request-circuit-breaker/275893/10 "2021-06-18T06:18:05Z")
> [@DavidTurner](#):
>
> The big change in newer versions is described in this blog post:
Unfortunately, upgrade to 7.x is just on the roadmap at this point given that it does not support a transport client and also getting away from the types. So, we have to continue with this version for a while. I hope Elastic continues to support their customers with version 6.8.
If there is an alternative to protect the cluster for large cardinality requests (buckets explosion), please let me know.
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [June 18, 2021, 6:53am UTC](https://discuss.elastic.co/t/elasticsearch-request-circuit-breaker/275893/11 "2021-06-18T06:53:54Z")
> [@animageofmine](#):
>
> Unfortunately, upgrade to 7.x is just on the roadmap at this point given that it does not support a transport client and also getting away from the types.
This is inaccurate:
- The main change in the [removal of mapping types](https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html) already happened, in 6.0. The only changes in this area in subsequent versions relate to adjusting any APIs that unnecessarily mention the (now-unique) type.
- The [Java transport client](https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/transport-client.html) is supported in all 7.x versions. It's deprecated, indicating our intention to remove support for it in 8.0, but that doesn't affect anything in 7.x.
---
### Author:  [@animageofmine](https://discuss.elastic.co/u/animageofmine)
#### Post date: [June 18, 2021, 6:50pm UTC](https://discuss.elastic.co/t/elasticsearch-request-circuit-breaker/275893/12 "2021-06-18T18:50:24Z")
> [@DavidTurner](#):
>
> - The main change in the [removal of mapping types](https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html) already happened, in 6.0. The only changes in this area in subsequent versions relate to adjusting any APIs that unnecessarily mention the (now-unique) type.
> - The [Java transport client](https://www.elastic.co/guide/en/elasticsearch/client/java-api/current/transport-client.html) is supported in all 7.x versions. It's deprecated, indicating our intention to remove support for it in 8.0, but that doesn't affect anything in 7.x.
Here's what I read for types:
> Indices created in Elasticsearch 7.0.0 or later no longer accept a `_default_` mapping. Indices created in 6.x will continue to function as before in Elasticsearch 6.x. Types are deprecated in APIs in 7.0, with breaking changes to the index creation, put mapping, get mapping, put template, get template and get field mappings APIs.
We are using the default type in 6.8, so got to be working.
We are using elastic4s Scala client for APIs and they stopped supporting updates to TCP client, so we need to update it.
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [June 19, 2021, 5:55am UTC](https://discuss.elastic.co/t/elasticsearch-request-circuit-breaker/275893/13 "2021-06-19T05:55:00Z")
> [@animageofmine](#):
>
> Here's what I read for types:
Yes that sounds about right. There's no need for a `_default_` mapping now that indices only have one type, nor is there any need to mention types in the APIs. These don't need any structural changes to adopt, you've already done the hard bit of moving to single-type indices in 6.x.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 17, 2021, 5:55am UTC](https://discuss.elastic.co/t/elasticsearch-request-circuit-breaker/275893/14 "2021-07-17T05:55:01Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.