] would be larger than configured breaker: 10213706956 [9.5gb], breaking
But what I don't understand is why does it affect even a simple root query ([http://localhost:9200/](http://localhost:9200/)). Shouldn't breakers stop just the query which is over the allowed size? Why the estimated size grows linearly and denies all subsequent requests on the affected node(s) after some minutes?
Do I misunderstand the concept of breakers entirely?
---
### Author:  [@dadoonet](https://discuss.elastic.co/u/dadoonet)
#### Post date: [November 24, 2017, 10:28pm UTC](https://discuss.elastic.co/t/circuit-breaker-always-trips/109067/6 "2017-11-24T22:28:19Z")
I don’t know.
But for sure, I saw in you mappings that you disabled doc\_values. Was there a reason for doing so?
---
### Author:  [@Attila\_Nagy](https://discuss.elastic.co/u/Attila_Nagy)
#### Post date: [November 24, 2017, 10:30pm UTC](https://discuss.elastic.co/t/circuit-breaker-always-trips/109067/7 "2017-11-24T22:30:53Z")
Nothing more than saving space where we don't use them.
---
### Author:  [@Attila\_Nagy](https://discuss.elastic.co/u/Attila_Nagy)
#### Post date: [November 25, 2017, 12:51am UTC](https://discuss.elastic.co/t/circuit-breaker-always-trips/109067/8 "2017-11-25T00:51:53Z")
OK, so the questions seem to be:
1. why does the request breakers' size constantly grow until it hits the limit and no query can get in? it's not a single query, but a bunch of queries (hence the steep curve on the graph, but it's not like a dirac delta).
"breakers" : {
"request" : {
"limit\_size\_in\_bytes" : 10187558092,
"limit\_size" : "9.4gb",
"estimated\_size\_in\_bytes" : 11343200256,
"estimated\_size" : "10.5gb",
"overhead" : 1.0,
"tripped" : 250
},
2. how could I see what queries account for these values?
3. how can it be that the JVM heap size (reported by elastic) doesn't show this increased memory usage? (maybe a counter leak, it doesn't get decremented after a query finishes?)
4. why is this localized to two nodes only (i will try to narrow it down to given indices, but it seems it happens only where the primary and replica shards of a given index are)
Also I think that the log/returned error (currently: [2017-11-24T21:52:55,425][WARN][o.e.i.b.request] [request] New used memory 10664646024 [9.9gb] for data of [\] would be larger than configured breaker: 10213706956 [9.5gb], breaking) should also include that individual query's estimated memory requirement, so if the estimated size grows by gigabytes every seconds, it may be traced better from the logs.
BTW, this is very similar to [https://github.com/elastic/elasticsearch/issues/26943](https://github.com/elastic/elasticsearch/issues/26943), except that I have a very rapid explosion in request breaker's size. (and doesn't have any plugins apart from icu)
---
### Author:  [@Attila\_Nagy](https://discuss.elastic.co/u/Attila_Nagy)
#### Post date: [November 25, 2017, 8:30am UTC](https://discuss.elastic.co/t/circuit-breaker-always-trips/109067/9 "2017-11-25T08:30:52Z")
OK, I've raised the breakers' limit to 2^63-1 bytes, it made the cluster stable.
So it seems this is a counter bug/leak, obviously no node can use 170 GiB of heap.
And I think I found the root cause.
Opened issue:
>
---
### Author:  [@dakrone](https://discuss.elastic.co/u/dakrone)
#### Post date: [November 29, 2017, 3:19am UTC](https://discuss.elastic.co/t/circuit-breaker-always-trips/109067/10 "2017-11-29T03:19:09Z")
Yep, that's definitely it. It's an issue where the stream is not closed, so the breaker isn't decrementing the accounting by the number of bytes. There's a PR open now to fix this.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [December 27, 2017, 3:19am UTC](https://discuss.elastic.co/t/circuit-breaker-always-trips/109067/11 "2017-12-27T03:19:14Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.