java.lang.OutOfMemoryError Kubernetes GKE

ritasker · January 15, 2018, 12:01pm

Morning All,

I am running Elastricsearch & Kibana with Fluentd in Kubernetes on the GKE.
Each node in the cluster is running kubernetes 1.8.5 and has 4 vCPUs, 15 GB memory.

I am running ES v6.0.0 with the following config.

cluster.name: "kubernetes-cluster"
network.host: 0.0.0.0
discovery.zen.minimum_master_nodes: 1
http.cors.enabled: true
http.cors.allow-origin: "*"

And the following JVM arguments

-XX:+UseConcMarkSweepGC, 
-XX:CMSInitiatingOccupancyFraction=75, 
-XX:+UseCMSInitiatingOccupancyOnly, 
-XX:+AlwaysPreTouch, 
-Xss1m, 
-Djava.awt.headless=true, 
-Dfile.encoding=UTF-8, 
-Djna.nosys=true, 
-XX:-OmitStackTraceInFastThrow, 
-Dio.netty.noUnsafe=true, 
-Dio.netty.noKeySetOptimization=true, 
-Dio.netty.recycler.maxCapacityPerThread=0, 
-Dlog4j.shutdownHookEnabled=false, 
-Dlog4j2.disable.jmx=true, 
-XX:+HeapDumpOnOutOfMemoryError, 
-Des.cgroups.hierarchy.override=/, 
-Xms2048m, -Xmx2048m, 
-Des.path.home=/usr/share/elasticsearch, 
-Des.path.conf=/usr/share/elasticsearch/config

Occasionally the ES container with get stuck in a crash loop and keep restarting. Going through the logs the exception below is thrown just before ES crashes and restarts.

gist.github.com

https://gist.github.com/ritasker/9c13f437324e76d1f3a7d602671ea198

Elasticsearch.txt

[2018-01-15T05:57:03,292][WARN ][o.e.m.j.JvmGcMonitorService] [_3m0Rmo] [gc][169] overhead, spent [2.4s] collecting in the last [2.4s]
[2018-01-15T05:57:32,682][WARN ][o.e.m.j.JvmGcMonitorService] [_3m0Rmo] [gc][170] overhead, spent [29.3s] collecting in the last [1.5s]
Dumping heap to java_pid1.hprof ...
java.lang.OutOfMemoryError: Java heap space
Heap dump file created [2392178503 bytes in 8.593 secs]
[2018-01-15T06:00:03,297][WARN ][o.e.m.j.JvmGcMonitorService] [_3m0Rmo] [gc][171] overhead, spent [2.3m] collecting in the last [2.6m]
[2018-01-15T06:00:09,991][WARN ][o.e.m.j.JvmGcMonitorService] [_3m0Rmo] [gc][173] overhead, spent [5s] collecting in the last [5s]
[2018-01-15T06:01:18,987][WARN ][o.e.m.j.JvmGcMonitorService] [_3m0Rmo] [gc][174] overhead, spent [1.1m] collecting in the last [28.8s]
[2018-01-15T06:02:43,695][WARN ][o.e.m.j.JvmGcMonitorService] [_3m0Rmo] [gc][175] overhead, spent [1.4m] collecting in the last [2m]
[2018-01-15T06:05:29,271][WARN ][o.e.t.TransportService   ] [_3m0Rmo] Received response for a request that has timed out, sent [309483ms] ago, timed out [293806ms] ago, action [cluster:monitor/nodes/stats[n]], node [{_3m0Rmo}{_3m0RmoDQ32a3N4T9kW3wA}{0H-s7v3FSlKAF-9Kyp5ItA}{10.0.0.21}{10.0.0.21:9300}], id [967]

This file has been truncated. show original

I hit the character count limit so put the exception in a gist.

Please let me know if more info is needed.

-- Rich

val · January 15, 2018, 3:49pm

You seem to be having very long GC cycles, so you're clearly short of heap. Since you have 15GB of RAM on each node, you can try to augment your heap to 7GB instead of 2GB and see if it works better.

system · February 12, 2018, 3:50pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.