Our cluster always execute some aggregations queries , in order to keep the memory usable, indices.breaker.request.limit set to 10% of the heap space, others like this:
indices.breaker.fielddata.limit: 15%
indices.breaker.total.limit: 20%
indices.fielddata.cache.size: 15%
but when query reach the limit , and query breaking .... it's still cause the heap space grow at 100% . and there is no OOM occur. the cluster is no response. and index rate keeps falling . how can i solve the problem?
ES_JAVA_OPTS
-Xmn5g -Xms15g -Xmx15g
when the problem happen, look at the jvm usage .........emmmm..........keeps 100%
jstat -gcutil -h10 16921 1000 999
S0 S1 E O M CCS YGC YGCT FGC FGCT GCT
100.00 0.00 100.00 100.00 91.88 82.11 9793 327.515 345 361.468 688.984
100.00 0.00 100.00 100.00 91.88 82.11 9793 327.515 345 361.468 688.984
100.00 0.00 100.00 100.00 91.88 82.11 9793 327.515 345 361.468 688.984
100.00 0.00 100.00 100.00 91.88 82.11 9793 327.515 345 361.468 688.984
Elasticsearch Logs:
[2017-12-28T14:27:11,773][WARN ][o.e.i.b.request ] [request] New used memory 1574091440 [1.4gb] for data of [<reused_arrays>] would be larger than configured breaker: 1543503872 [1.4gb], breaking
[2017-12-28T14:27:13,280][WARN ][o.e.i.b.request ] [request] New used memory 1710013784 [1.5gb] for data of [<reused_arrays>] would be larger than configured breaker: 1543503872 [1.4gb], breaking
[2017-12-28T14:27:57,322][WARN ][o.e.m.j.JvmGcMonitorService] [es-honey-001.yz.momo.com] [gc][63734] overhead, spent [1.1s] collecting in the last [1.4s]
[2017-12-28T14:28:02,860][WARN ][o.e.m.j.JvmGcMonitorService] [es-honey-001.yz.momo.com] [gc][young][63736][9792] duration [4s], collections [1]/[4.5s], total [4s]/[5.4m], memory [10gb]->[10.5gb]/[14.3gb], all_pools {[young] [2.3gb]->[5.1mb]/[3.7gb]}{[survivor] [640mb]->[640mb]/[640mb]}{[old] [7gb]->[9.8gb]/[10gb]}
[2017-12-28T14:28:02,860][WARN ][o.e.m.j.JvmGcMonitorService] [es-honey-001.yz.momo.com] [gc][63736] overhead, spent [4.1s] collecting in the last [4.5s]
[2017-12-28T14:28:29,196][WARN ][o.e.m.j.JvmGcMonitorService] [es-honey-001.yz.momo.com] [gc][old][63739][162] duration [23.8s], collections [2]/[24.3s], total [23.8s]/[46.4s], memory [13.7gb]->[11.4gb]/[14.3gb], all_pools {[young] [3.1gb]->[1.4gb]/[3.7gb]}{[survivor] [640mb]->[0b]/[640mb]}{[old] [9.8gb]->[10gb]/[10gb]}
Elasticsearch version (bin/elasticsearch --version): elasticsearch-5.5.3
Plugins installed: [none]
JVM version (java -version): java-1.8.0-oracle
OS version : Linux xxxx 3.10.0-327.22.2.el7.x86_64 CentOS Linux release 7.2.1511