When I query data in past 7 days in Kibana (search on 1 dashboard contains 9 visualization panels, including aggregations, and descending size are about 10~20) I got these error:
[2017-09-21T03:08:24,180][WARN ][o.e.m.j.JvmGcMonitorService] [master-2] [gc][382351] overhead, spent [777ms] collecting in the last [1s]
[2017-09-21T03:08:26,324][WARN ][o.e.m.j.JvmGcMonitorService] [master-2] [gc][382352] overhead, spent [2s] collecting in the last [2.1s]
[2017-09-21T03:08:27,381][WARN ][o.e.m.j.JvmGcMonitorService] [master-2] [gc][382353] overhead, spent [809ms] collecting in the last [1s]
[2017-09-21T03:08:27,566][WARN ][o.e.i.b.request ] [request] New used memory 7480688384 [6.9gb] for data of [<reused_arrays>] would be larger t
han configured breaker: 6400612761 [5.9gb], breaking
[2017-09-21T03:08:28,381][INFO ][o.e.m.j.JvmGcMonitorService] [master-2] [gc][382354] overhead, spent [307ms] collecting in the last [1s]
[2017-09-21T03:10:21,764][WARN ][o.e.m.j.JvmGcMonitorService] [master-2] [gc][382466] overhead, spent [2s] collecting in the last [2.3s]
[2017-09-21T03:10:22,811][WARN ][o.e.m.j.JvmGcMonitorService] [master-2] [gc][382467] overhead, spent [681ms] collecting in the last [1s]
[2017-09-21T03:10:23,812][INFO ][o.e.m.j.JvmGcMonitorService] [master-2] [gc][382468] overhead, spent [385ms] collecting in the last [1s]
[2017-09-21T03:10:49,816][WARN ][o.e.m.j.JvmGcMonitorService] [master-2] [gc][382494] overhead, spent [559ms] collecting in the last [1s]
[2017-09-21T03:10:51,542][WARN ][o.e.m.j.JvmGcMonitorService] [master-2] [gc][382495] overhead, spent [1.6s] collecting in the last [1.7s]
[2017-09-21T03:10:52,720][WARN ][o.e.m.j.JvmGcMonitorService] [master-2] [gc][382496] overhead, spent [942ms] collecting in the last [1.1s]
[2017-09-21T03:10:57,807][WARN ][o.e.m.j.JvmGcMonitorService] [master-2] [gc][382497] overhead, spent [2.9s] collecting in the last [3s]
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid21870.hprof ...
Heap dump file created [10780134241 bytes in 61.448 secs]
[2017-09-21T03:12:15,804][ERROR][o.e.x.m.c.i.IndexStatsCollector] [master-2] collector [index-stats] timed out when collecting data
[2017-09-21T03:12:16,026][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [master-2] fatal error in thread [elasticsearch[master-2][search][T#5]], exiting
java.lang.OutOfMemoryError: Java heap space
at org.elasticsearch.common.util.PageCacheRecycler$1.newInstance(PageCacheRecycler.java:99) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.util.PageCacheRecycler$1.newInstance(PageCacheRecycler.java:96) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.recycler.DequeRecycler.obtain(DequeRecycler.java:53) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.recycler.AbstractRecycler.obtain(AbstractRecycler.java:33) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.recycler.DequeRecycler.obtain(DequeRecycler.java:28) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.recycler.FilterRecycler.obtain(FilterRecycler.java:39) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.recycler.Recyclers$3.obtain(Recyclers.java:119) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.recycler.FilterRecycler.obtain(FilterRecycler.java:39) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.util.PageCacheRecycler.bytePage(PageCacheRecycler.java:147) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.util.AbstractBigArray.newBytePage(AbstractBigArray.java:112) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.util.BigByteArray.<init>(BigByteArray.java:44) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.util.BigArrays.newByteArray(BigArrays.java:464) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.util.BigArrays.resize(BigArrays.java:488) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.common.util.BigArrays.grow(BigArrays.java:502) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.search.aggregations.metrics.cardinality.HyperLogLogPlusPlus.ensureCapacity(HyperLogLogPlusPlus.java:197) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.search.aggregations.metrics.cardinality.HyperLogLogPlusPlus.collect(HyperLogLogPlusPlus.java:232) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.search.aggregations.metrics.cardinality.CardinalityAggregator$OrdinalsCollector.postCollect(CardinalityAggregator.java:280) ~[elasticsearch-5.4.0.jar:5.4.0]
at org.elasticsearch.search.aggregations.metrics.cardinality.CardinalityAggregator.postCollectLastCollector(CardinalityAggregator.java:120) ~[elasticsearch-5.4.0.jar:5.4.0]
at ...
After that some of my data nodes throw out 'java.lang.OutOfMemoryError: Java heap space' then lost service.
There are service logs in my ES and index splited by day .
Mem: total 31G
jvm.options:
-Xms16g
-Xmx16g
And 10 data nodes and 25 indices,164 shards.
Anyone helps?
