# Finding out cause of circuit breaking exception (Data too large)

**URL:** <https://discuss.elastic.co/t/finding-out-cause-of-circuit-breaking-exception-data-too-large/351217>\
**Category:** Elasticsearch\
**Created:** [January 17, 2024, 2:49am UTC](https://discuss.elastic.co/t/finding-out-cause-of-circuit-breaking-exception-data-too-large/351217 "2024-01-17T02:49:59Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![hjazz6](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hjazz6/32/79007_2.png) [@hjazz6](https://discuss.elastic.co/u/hjazz6)\
**Post date:** [January 17, 2024, 2:49am UTC](https://discuss.elastic.co/t/finding-out-cause-of-circuit-breaking-exception-data-too-large/351217/1 "2024-01-17T02:49:59Z")

</div>

Hi,

I'm getting the error below occasionally, and I'm trying to find the cause of it so I can try to resolve it.

`circuit_breaking_exception: [parent] Data too large, data for [<http_request>] would be [32620387928/30.3gb], which is larger than the limit of [31621696716/29.4gb], real usage: [32620387928/30.3gb], new bytes reserved: [0/0b], usages [eql_sequence=0/0b, fielddata=24272253421/22.6gb, request=0.0b, inflight_requests=0/0b, model_inference=0.0b]`

I'm running ES v8.8.0, with multiple instances of filebeat writing to a single index, at the indexing rate of about 160K/s. The ES cluster is on Kubernetes, with 3 master ES nodes and 40 data nodes (1.3TB of memory (usually at 50-60% usage), 126TB/600TB of SSD space free). Once I hit the error message above, I won't be able to access the Kibana/ES anymore, and would have to restart the ES nodes.

One strange thing I just happened to notice was when I was got the error, the indexing rate of the index currently being written to got up to an impossibly high number, like 9-digit (on the Stack Monitoring --\> Indices page, time window: last 2 min). Then it drops to 2+M, which is still unusually high since I'm expecting at most ~160K/s. Not sure if this has anything to do with the error.

Is there a way to find out what is causing this issue, such as some complex query running, or the ingest rate is too high (I have gotten this error at a lower indexing rate of ~100K/s too), etc?

So far, I've noticed that the `real usage` is not that much higher than the `limit`. Would increasing the `limit` from `29.4gb` to, say, `32gb` be possible and feasible?

Thank you.

---

<div class="post-metadata">

**Author:** ![carly.richmond](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carly.richmond/32/104935_2.png) [@carly.richmond](https://discuss.elastic.co/u/carly.richmond)\
**Post date:** [January 22, 2024, 4:59pm UTC](https://discuss.elastic.co/t/finding-out-cause-of-circuit-breaking-exception-data-too-large/351217/2 "2024-01-22T16:59:19Z")

</div>

Hi @hjazz6,

Thanks for sharing. Are you using fielddata on text fields at all [as per the documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/circuit-breaker-errors.html)? It might be worth checking the state of the breakers using the [`stats` API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-stats.html).

Feel free to share the output and your cluster settings and we can see if we can help further.

---

<div class="post-metadata">

**Author:** ![hjazz6](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hjazz6/32/79007_2.png) [@hjazz6](https://discuss.elastic.co/u/hjazz6)\
**Post date:** [January 23, 2024, 1:56am UTC](https://discuss.elastic.co/t/finding-out-cause-of-circuit-breaking-exception-data-too-large/351217/3 "2024-01-23T01:56:27Z")

</div>

Hi @carly.richmond ,

Thanks for replying.

Out of the 102 fields I have, 9 are "text" fields.

I'm not sure how to interpret the output of `GET _nodes/stats/breaker`, and I can't copy and paste the entire output as my ES is on another (offline) machine. Perhaps you can let me know which fields I should be looking at?

Here is a portion of the output (of one of the nodes):

```auto
"attributes": {
    ...
    "ml.allocated_processors_double": "12.0",
    "data": "warm",
    "ml.allocated_processors": "12",
    "ml.machine_memory": "68719476736",
    "ml.max_jvm_size": "33285996544"
},
"breakers": {
    "eql_sequence": {
        "limit_size_in_bytes": 16642998272,
        "limit_size": "15.5gb",
        "estimated_size_in_bytes": 0,
        "estimated_size": "0b",
        "overhead": 1,
        "tripped": 0
    },
    "fielddata": {
        "limit_size_in_bytes": 13314398617,
        "limit_size": "12.3gb",
        "estimated_size_in_bytes": 0,
        "estimated_size": "0b",
        "overhead": 1.03,
        "tripped": 0
    },
    "request": {
        "limit_size_in_bytes": 19971597926,
        "limit_size": "18.5gb",
        "estimated_size_in_bytes": 0,
        "estimated_size": "0b",
        "overhead": 1,
        "tripped": 0
    },
    "inflight_requests": {
        "limit_size_in_bytes": 33285996544,
        "limit_size": "31gb",
        "estimated_size_in_bytes": 7412747,
        "estimated_size": "7mb",
        "overhead": 2,
        "tripped": 0
    },
    "model_inference": {
        "limit_size_in_bytes": 16642998272,
        "limit_size": "15.5gb",
        "estimated_size_in_bytes": 0,
        "estimated_size": "0b",
        "overhead": 1,
        "tripped": 0
    },
    "parent": {
        "limit_size_in_bytes": 31621696716,
        "limit_size": "29.4gb",
        "estimated_size_in_bytes": 16919285712,
        "estimated_size": "15.7gb",
        "overhead": 1,
        "tripped": 0
    }
}

```

All `tripped` values of all nodes are `0`.

One observation I made was when the `circuit breaking exception` happened yesterday again, the JVM heap memory usage was at least 85%.

---

<div class="post-metadata">

**Author:** ![carly.richmond](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carly.richmond/32/104935_2.png) [@carly.richmond](https://discuss.elastic.co/u/carly.richmond)\
**Post date:** [January 23, 2024, 9:53am UTC](https://discuss.elastic.co/t/finding-out-cause-of-circuit-breaking-exception-data-too-large/351217/4 "2024-01-23T09:53:53Z")

</div>

Hi @hjazz6,

It would be looking at the `tripped` attribute for the breakers to find the number of times it has been triggered: If you check out the breakers property [further down in the documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-nodes-stats.html#:~:text=breakers,of%20memory%20error.) a description of each field is given.

> [\<http\_request\>] would be [32620387928/30.3gb], which is larger than the limit of [31621696716/29.4gb]

The circuit breaking exception is an indication of JVM pressure so it does make sense that you observed high usage. Changing the limits isn't really recommended as they are designed to prevent OOM errors.

Do you know what kind of request was being processed when you saw the circuit breaker trigger yesterday? It might be worth checking the Elasticsearch logs and potentially the slow log to see what activity was happening around that time? Then you can see if you can split the request.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 20, 2024, 9:54am UTC](https://discuss.elastic.co/t/finding-out-cause-of-circuit-breaking-exception-data-too-large/351217/5 "2024-02-20T09:54:53Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

---

<div class="post-metadata">

**Author:** ![Stef\_Nestor](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stef_nestor/32/85483_2.png) [@Stef\_Nestor](https://discuss.elastic.co/u/Stef_Nestor)\
**Post date:** [December 4, 2024, 9:36pm UTC](https://discuss.elastic.co/t/finding-out-cause-of-circuit-breaking-exception-data-too-large/351217/6 "2024-12-04T21:36:25Z")

</div>

This article is a popular google search result. For more information, see also

- [Managing and troubleshooting Elasticsearch memory](https://www.elastic.co/blog/managing-and-troubleshooting-elasticsearch-memory)
- [Troubleshooting Circuit breaker errors](https://www.elastic.co/guide/en/elasticsearch/reference/current/circuit-breaker-errors.html) which may surface as [Rejected ingest requests](https://www.elastic.co/guide/en/elasticsearch/reference/current/rejected-requests.html) especially during [Hot Spotting](https://www.elastic.co/guide/en/elasticsearch/reference/current/hotspotting.html)
- [Enable Monitoring](https://www.elastic.co/guide/en/elasticsearch/reference/current/monitoring-overview.html) and consider enabling [Slow Logs](https://www.elastic.co/guide/en/elasticsearch/reference/master/index-modules-slowlog.html) on top