Hi guys,
Here we are for a new topic again
I'm in trouble with mu Elastic Cluster :
Got on 2 nodes in particular, heap usage in red at all times. (Around 19.7 min & 19.8 Max)
We jus lost one node few minutes ago, and 2 mores few days ago too.
HEAP : XMS/XMX 20G / Node
128G Set on Each Node ( Only 20 for ES)
i don't realy what is the real cause of this..
Some logs here :
> in the last [17.3s]
> [2018-03-19T11:55:29,329][WARN ][o.e.m.j.JvmGcMonitorService] XXX] [gc][old][657803][29914] duration [18.1s], collections [1]/[18.4s], total [18.1s]/[1h], memory [19.7gb]->[19.7gb]/[19.8gb], all_pools {[young] [1.4gb]->[1.4gb]/[1.4gb]}{[survivor] [138.3mb]->[152.9mb]/[191.3mb]}{[old] [18.1gb]->[18.1gb]/[18.1gb]}
> [2018-03-19T11:55:29,329][WARN ][o.e.m.j.JvmGcMonitorService] XXX] [gc][657803] overhead, spent [18.1s] collecting in the last [18.4s]
> [2018-03-19T11:55:49,159][WARN ][o.e.m.j.JvmGcMonitorService] [XXX [gc][old][657804][29915] duration [19.7s], collections [1]/[19.9s], total [19.7s]/[1h], memory [19.7gb]->[19.7gb]/[19.8gb], all_pools {[young] [1.4gb]->[1.4gb]/[1.4gb]}{[survivor] [152.9mb]->[159.7mb]/[191.3mb]}{[old] [18.1gb]->[18.1gb]/[18.1gb]}
> [2018-03-19T11:55:49,164][WARN ][o.e.m.j.JvmGcMonitorService] [XXX] [gc][657804] overhead, spent [19.7s] collecting in the last [19.9s]
> [2018-03-19T12:46:38,130][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] XXX] fatal error in thread [I/O dispatcher 28], exiting
> java.lang.OutOfMemoryError: Java heap space
> [2018-03-19T12:46:21,040][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [xxx] fatal error in thread [elasticsearch[XXX][bulk][T#25]], exiting
> java.lang.OutOfMemoryError: Java heap space
> [2018-03-19T12:46:38,132][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [xxx] fatal error in thread [elasticsearch[XXX][bulk][T#24]], exiting
> java.lang.OutOfMemoryError: Java heap space
> [2018-03-19T12:46:52,659][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [xxx] fatal error in thread [elasticsearch[xxx][bulk][T#28]], exiting
> java.lang.OutOfMemoryError: Java heap space
> [2018-03-19T12:46:21,042][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [xxx] fatal error in thread [I/O dispatcher 31], exiting
> java.lang.OutOfMemoryError: Java heap space
> at java.util.HashMap$KeySet.iterator(Unknown Source) ~[?:1.8.0_73]
> at java.util.HashSet.iterator(Unknown Source) ~[?:1.8.0_73]
> at java.util.Collections$UnmodifiableCollection$1.<init>(Unknown Source) ~[?:1.8.0_73]
> at java.util.Collections$UnmodifiableCollection.iterator(Unknown Source) ~[?:1.8.0_73]
> at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:212) ~[?:?]
> at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280) ~[?:?]
> at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[?:?]
> at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588) ~[?:?]
> at java.lang.Thread.run(Unknown Source) [?:1.8.0_73]
If any one get any advice or idea about this problem i'm open to suggestion.