# Circuit breaker not effective

**URL:** <https://discuss.elastic.co/t/circuit-breaker-not-effective/98681>\
**Category:** Elasticsearch\
**Created:** [August 29, 2017, 12:17pm UTC](https://discuss.elastic.co/t/circuit-breaker-not-effective/98681 "2017-08-29T12:17:02Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![fphilippon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fphilippon/32/21614_2.png) [@fphilippon](https://discuss.elastic.co/u/fphilippon)\
**Post date:** [August 29, 2017, 12:17pm UTC](https://discuss.elastic.co/t/circuit-breaker-not-effective/98681/1 "2017-08-29T12:17:02Z")

</div>

Hi community!

We have a situation on our Elasticsearch cluster where a single request can quickly bring down all cluster nodes (with an OOM exception).

We updated the cluster configuration in order to have following circuits breakers in place:

"indices":{"breaker":{"fielddata":{"limit":"50%"},"request":{"limit":"20%"},"total":{"limit":"30%"}}}}

We also tried to set a size limit for the fielddata cache (40%) but still getting the OOM exception on this request.

Some tips about our cluster topology:

- 5 data nodes
- max heap : 8gb
- 286 indices
- 3072 shards
- 2,857,660,643 docs
- 4.07TB

elasticsearch node log with exception :

> <https://gist.github.com/fphilippon/2e8491d8b82b060d26f5ad891277b955>

detailed query :

> <https://gist.github.com/fphilippon/44122d2ae12fa146d6f039f34e32ba6f>

We would like to know how to make sure this kind of requests will not be able to crash our entire cluster and how to go further in the root cause analysis.

Many thanks for your help! 🙂

---

<div class="post-metadata">

**Author:** ![colings86](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/colings86/32/44960_2.png) [@colings86](https://discuss.elastic.co/u/colings86)\
**Post date:** [August 29, 2017, 12:27pm UTC](https://discuss.elastic.co/t/circuit-breaker-not-effective/98681/2 "2017-08-29T12:27:18Z")

</div>

Which version of Elasticsearch are you running? If you are running a version prior to 5.4.2 you may be running into these issues: [#25010](https://github.com/elastic/elasticsearch/pull/25010) and [#24941](https://github.com/elastic/elasticsearch/pull/24941). There are also still some known issues around aggregations and OOM which we are tracking in [#26012](https://github.com/elastic/elasticsearch/issues/26012)

---

<div class="post-metadata">

**Author:** ![Mark\_Harwood](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mark_harwood/32/10538_2.png) [@Mark\_Harwood](https://discuss.elastic.co/u/Mark_Harwood)\
**Post date:** [August 29, 2017, 12:34pm UTC](https://discuss.elastic.co/t/circuit-breaker-not-effective/98681/3 "2017-08-29T12:34:46Z")

</div>

> fielddata”:{“limit”:“50%”}

Fielddata is best avoided if you can use doc values instead. See [Support in the Wild: My Biggest Elasticsearch Problem at Scale | Elastic Blog](https://www.elastic.co/blog/support-in-the-wild-my-biggest-elasticsearch-problem-at-scale)

> [@fphilippon](#):
>
> 5 data nodes  
> 286 indices  
> 3072 shards

An average of \>10 shards per index and 5 data nodes? Having more shards than data nodes is useful if you plan on expanding out into more data nodes in future but otherwise it's a less efficient way to store the data.

---

<div class="post-metadata">

**Author:** ![fphilippon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fphilippon/32/21614_2.png) [@fphilippon](https://discuss.elastic.co/u/fphilippon)\
**Post date:** [August 29, 2017, 12:41pm UTC](https://discuss.elastic.co/t/circuit-breaker-not-effective/98681/4 "2017-08-29T12:41:49Z")

</div>

We are running the 5.3.1 version of Elasticsearch.  
We will plan soon an upgrade to 5.4.2 and see if we still have issues.

Thanks!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 26, 2017, 12:41pm UTC](https://discuss.elastic.co/t/circuit-breaker-not-effective/98681/5 "2017-09-26T12:41:56Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.