Old generation of the heap space stuck at 100% after request circuit breaker


#1

Our cluster always execute some aggregations queries , in order to keep the memory usable, indices.breaker.request.limit set to 10% of the heap space, others like this:
indices.breaker.fielddata.limit: 15%
indices.breaker.total.limit: 20%
indices.fielddata.cache.size: 15%
but when query reach the limit , and query breaking .... it's still cause the heap space grow at 100% . and there is no OOM occur. the cluster is no response. and index rate keeps falling . how can i solve the problem?

ES_JAVA_OPTS
-Xmn5g -Xms15g -Xmx15g

when the problem happen, look at the jvm usage .........emmmm..........keeps 100%

jstat -gcutil -h10 16921 1000 999
S0 S1 E O M CCS YGC YGCT FGC FGCT GCT
100.00 0.00 100.00 100.00 91.88 82.11 9793 327.515 345 361.468 688.984
100.00 0.00 100.00 100.00 91.88 82.11 9793 327.515 345 361.468 688.984
100.00 0.00 100.00 100.00 91.88 82.11 9793 327.515 345 361.468 688.984
100.00 0.00 100.00 100.00 91.88 82.11 9793 327.515 345 361.468 688.984

Elasticsearch Logs:

[2017-12-28T14:27:11,773][WARN ][o.e.i.b.request ] [request] New used memory 1574091440 [1.4gb] for data of [<reused_arrays>] would be larger than configured breaker: 1543503872 [1.4gb], breaking
[2017-12-28T14:27:13,280][WARN ][o.e.i.b.request ] [request] New used memory 1710013784 [1.5gb] for data of [<reused_arrays>] would be larger than configured breaker: 1543503872 [1.4gb], breaking
[2017-12-28T14:27:57,322][WARN ][o.e.m.j.JvmGcMonitorService] [es-honey-001.yz.momo.com] [gc][63734] overhead, spent [1.1s] collecting in the last [1.4s]
[2017-12-28T14:28:02,860][WARN ][o.e.m.j.JvmGcMonitorService] [es-honey-001.yz.momo.com] [gc][young][63736][9792] duration [4s], collections [1]/[4.5s], total [4s]/[5.4m], memory [10gb]->[10.5gb]/[14.3gb], all_pools {[young] [2.3gb]->[5.1mb]/[3.7gb]}{[survivor] [640mb]->[640mb]/[640mb]}{[old] [7gb]->[9.8gb]/[10gb]}
[2017-12-28T14:28:02,860][WARN ][o.e.m.j.JvmGcMonitorService] [es-honey-001.yz.momo.com] [gc][63736] overhead, spent [4.1s] collecting in the last [4.5s]
[2017-12-28T14:28:29,196][WARN ][o.e.m.j.JvmGcMonitorService] [es-honey-001.yz.momo.com] [gc][old][63739][162] duration [23.8s], collections [2]/[24.3s], total [23.8s]/[46.4s], memory [13.7gb]->[11.4gb]/[14.3gb], all_pools {[young] [3.1gb]->[1.4gb]/[3.7gb]}{[survivor] [640mb]->[0b]/[640mb]}{[old] [9.8gb]->[10gb]/[10gb]}

Elasticsearch version (bin/elasticsearch --version): elasticsearch-5.5.3

Plugins installed: [none]

JVM version (java -version): java-1.8.0-oracle

OS version : Linux xxxx 3.10.0-327.22.2.el7.x86_64 CentOS Linux release 7.2.1511


(Christian Dahlqvist) #2

What is your use case? What is the full output of the cluster stats API?


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.