[WARN ][o.e.m.j.JvmGcMonitorService] [esm3] [gc][475373] overhead, spent [658ms] collecting in the last [1.2s]

Elastic Stack Elasticsearch
1

I just noticed, my master nodes all producing following in logs:

esm3 | [2018-02-22T04:32:02,931][WARN ][o.e.m.j.JvmGcMonitorService] [esm3] [gc][475373] overhead, spent [658ms] collecting in the last [1.2s]

there are 3 master-eligible nodes, bellow is actual master node:

# top -n 1 | head
top - 06:06:42 up 5 days, 13:42,  2 users,  load average: 0.31, 0.40, 0.35
Tasks:  93 total,   1 running,  92 sleeping,   0 stopped,   0 zombie
%Cpu(s):  5.2 us,  0.4 sy,  0.0 ni, 94.0 id,  0.0 wa,  0.0 hi,  0.4 si,  0.0 st
KiB Mem :  7663592 total,  4881024 free,  1861236 used,   921332 buff/cache
KiB Swap:        0 total,        0 free,        0 used.  5512280 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                             
 1376 alexus    20   0 3730832 1.479g  27460 S  13.3 20.2 891:58.28 java                                
  647 root      20   0  691908 116228  43416 S   6.7  1.5  40:21.04 dockerd                             
    1 root      20   0   56964   6656   5164 S   0.0  0.1   0:02.92 systemd                             
#

I used ES_HEAP_SIZE=4g for elasticsearch (master-eligible nodes).

Please advise.

2

master node (ES_HEAP_SIZE=4g):

esm1             | Feb 22, 2018 5:05:33 PM com.google.api.client.http.HttpRequest execute
esm1             | WARNING: exception thrown while executing request
esm1             | java.net.SocketTimeoutException: Read timed out

ingest node (ES_HEAP_SIZE=16g):

elasticsearch | Caused by: java.lang.OutOfMemoryError: Java heap space
elasticsearch | [2018-02-22T00:36:53,662][ERROR][i.n.u.ResourceLeakDetector] LEAK: ByteBuf.release() was not called before it's garbage-collected. Enable advanced leak reporting to find out where the leak
+occurred. To enable advanced leak reporting, specify the JVM option '-Dio.netty.leakDetection.level=advanced' or call ResourceLeakDetector.setLevel() See http://netty.io/wiki/reference-counted-objects.html for
+more information.
elasticsearch | [2018-02-22T00:36:53,641][WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [elasticsearch] exception caught on transport layer [NettyTcpChannel{localAddress=/10.0.0.15:9300,
+remoteAddress=/10.0.0.11:42982}], closing connection
elasticsearch | java.lang.IllegalArgumentException: Self-suppression not permitted
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.