# Old generation of the heap space stuck at 100% after request circuit breaker

**URL:** <https://discuss.elastic.co/t/old-generation-of-the-heap-space-stuck-at-100-after-request-circuit-breaker/113566>\
**Category:** Elasticsearch\
**Created:** [December 29, 2017, 10:35am UTC](https://discuss.elastic.co/t/old-generation-of-the-heap-space-stuck-at-100-after-request-circuit-breaker/113566 "2017-12-29T10:35:09Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![dumplings](https://avatars.discourse-cdn.com/v4/letter/d/a4c791/32.png) [@dumplings](https://discuss.elastic.co/u/dumplings)\
**Post date:** [December 29, 2017, 10:35am UTC](https://discuss.elastic.co/t/old-generation-of-the-heap-space-stuck-at-100-after-request-circuit-breaker/113566/1 "2017-12-29T10:35:09Z")

</div>

Our cluster always execute some aggregations queries , in order to keep the memory usable, indices.breaker.request.limit set to 10% of the heap space, others like this:  
indices.breaker.fielddata.limit: 15%  
indices.breaker.total.limit: 20%  
indices.fielddata.cache.size: 15%  
but when query reach the limit , and query breaking .... it's still cause the heap space grow at 100% . and there is no OOM occur. the cluster is no response. and index rate keeps falling . how can i solve the problem?

ES\_JAVA\_OPTS  
-Xmn5g -Xms15g -Xmx15g

when the problem happen, look at the jvm usage .........emmmm..........keeps 100%

```auto
jstat -gcutil -h10 16921 1000 999
S0 S1 E O M CCS YGC YGCT FGC FGCT GCT
100.00 0.00 100.00 100.00 91.88 82.11 9793 327.515 345 361.468 688.984
100.00 0.00 100.00 100.00 91.88 82.11 9793 327.515 345 361.468 688.984
100.00 0.00 100.00 100.00 91.88 82.11 9793 327.515 345 361.468 688.984
100.00 0.00 100.00 100.00 91.88 82.11 9793 327.515 345 361.468 688.984

```

Elasticsearch Logs:

```auto
[2017-12-28T14:27:11,773][WARN][o.e.i.b.request] [request] New used memory 1574091440 [1.4gb] for data of [<reused_arrays>] would be larger than configured breaker: 1543503872 [1.4gb], breaking
[2017-12-28T14:27:13,280][WARN][o.e.i.b.request] [request] New used memory 1710013784 [1.5gb] for data of [<reused_arrays>] would be larger than configured breaker: 1543503872 [1.4gb], breaking
[2017-12-28T14:27:57,322][WARN][o.e.m.j.JvmGcMonitorService] [es-honey-001.yz.momo.com] [gc][63734] overhead, spent [1.1s] collecting in the last [1.4s]
[2017-12-28T14:28:02,860][WARN][o.e.m.j.JvmGcMonitorService] [es-honey-001.yz.momo.com] [gc][young][63736][9792] duration [4s], collections [1]/[4.5s], total [4s]/[5.4m], memory [10gb]->[10.5gb]/[14.3gb], all_pools {[young] [2.3gb]->[5.1mb]/[3.7gb]}{[survivor] [640mb]->[640mb]/[640mb]}{[old] [7gb]->[9.8gb]/[10gb]}
[2017-12-28T14:28:02,860][WARN][o.e.m.j.JvmGcMonitorService] [es-honey-001.yz.momo.com] [gc][63736] overhead, spent [4.1s] collecting in the last [4.5s]
[2017-12-28T14:28:29,196][WARN][o.e.m.j.JvmGcMonitorService] [es-honey-001.yz.momo.com] [gc][old][63739][162] duration [23.8s], collections [2]/[24.3s], total [23.8s]/[46.4s], memory [13.7gb]->[11.4gb]/[14.3gb], all_pools {[young] [3.1gb]->[1.4gb]/[3.7gb]}{[survivor] [640mb]->[0b]/[640mb]}{[old] [9.8gb]->[10gb]/[10gb]}

```

Elasticsearch version (bin/elasticsearch --version): elasticsearch-5.5.3

Plugins installed: [none]

JVM version (java -version): java-1.8.0-oracle

OS version : Linux xxxx 3.10.0-327.22.2.el7.x86\_64 CentOS Linux release 7.2.1511

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [December 31, 2017, 10:00am UTC](https://discuss.elastic.co/t/old-generation-of-the-heap-space-stuck-at-100-after-request-circuit-breaker/113566/2 "2017-12-31T10:00:24Z")

</div>

What is your use case? What is the full output of the [cluster stats API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-stats.html)?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 28, 2018, 10:00am UTC](https://discuss.elastic.co/t/old-generation-of-the-heap-space-stuck-at-100-after-request-circuit-breaker/113566/3 "2018-01-28T10:00:25Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.