Elasticsearch Over Heap Any times

Hi guys,

Here we are for a new topic again :stuck_out_tongue:

I'm in trouble with mu Elastic Cluster :

Got on 2 nodes in particular, heap usage in red at all times. (Around 19.7 min & 19.8 Max)

We jus lost one node few minutes ago, and 2 mores few days ago too.

HEAP : XMS/XMX 20G / Node

128G Set on Each Node ( Only 20 for ES)

i don't realy what is the real cause of this..

Some logs here :

>  in the last [17.3s]
> [2018-03-19T11:55:29,329][WARN ][o.e.m.j.JvmGcMonitorService] XXX] [gc][old][657803][29914] duration [18.1s], collections [1]/[18.4s], total [18.1s]/[1h], memory [19.7gb]->[19.7gb]/[19.8gb], all_pools {[young] [1.4gb]->[1.4gb]/[1.4gb]}{[survivor] [138.3mb]->[152.9mb]/[191.3mb]}{[old] [18.1gb]->[18.1gb]/[18.1gb]}
> [2018-03-19T11:55:29,329][WARN ][o.e.m.j.JvmGcMonitorService] XXX] [gc][657803] overhead, spent [18.1s] collecting in the last [18.4s]
> [2018-03-19T11:55:49,159][WARN ][o.e.m.j.JvmGcMonitorService] [XXX [gc][old][657804][29915] duration [19.7s], collections [1]/[19.9s], total [19.7s]/[1h], memory [19.7gb]->[19.7gb]/[19.8gb], all_pools {[young] [1.4gb]->[1.4gb]/[1.4gb]}{[survivor] [152.9mb]->[159.7mb]/[191.3mb]}{[old] [18.1gb]->[18.1gb]/[18.1gb]}
> [2018-03-19T11:55:49,164][WARN ][o.e.m.j.JvmGcMonitorService] [XXX] [gc][657804] overhead, spent [19.7s] collecting in the last [19.9s]
> [2018-03-19T12:46:38,130][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] XXX] fatal error in thread [I/O dispatcher 28], exiting
> java.lang.OutOfMemoryError: Java heap space
> [2018-03-19T12:46:21,040][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [xxx] fatal error in thread [elasticsearch[XXX][bulk][T#25]], exiting
> java.lang.OutOfMemoryError: Java heap space
> [2018-03-19T12:46:38,132][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [xxx] fatal error in thread [elasticsearch[XXX][bulk][T#24]], exiting
> java.lang.OutOfMemoryError: Java heap space
> [2018-03-19T12:46:52,659][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [xxx] fatal error in thread [elasticsearch[xxx][bulk][T#28]], exiting
> java.lang.OutOfMemoryError: Java heap space
> [2018-03-19T12:46:21,042][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [xxx] fatal error in thread [I/O dispatcher 31], exiting
> java.lang.OutOfMemoryError: Java heap space
>         at java.util.HashMap$KeySet.iterator(Unknown Source) ~[?:1.8.0_73]
>         at java.util.HashSet.iterator(Unknown Source) ~[?:1.8.0_73]
>         at java.util.Collections$UnmodifiableCollection$1.<init>(Unknown Source) ~[?:1.8.0_73]
>         at java.util.Collections$UnmodifiableCollection.iterator(Unknown Source) ~[?:1.8.0_73]
>         at org.apache.http.impl.nio.reactor.BaseIOReactor.validate(BaseIOReactor.java:212) ~[?:?]
>         at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:280) ~[?:?]
>         at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104) ~[?:?]
>         at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588) ~[?:?]
>         at java.lang.Thread.run(Unknown Source) [?:1.8.0_73]

If any one get any advice or idea about this problem i'm open to suggestion.

Which version of Elasticsearch are you using? What is the output of the cluster stats API?

I'm running V5.5.1

Nothing special, cluster jsut backed green so all fine for now

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.