I just noticed, my master nodes all producing following in logs:
esm3 | [2018-02-22T04:32:02,931][WARN ][o.e.m.j.JvmGcMonitorService] [esm3] [gc][475373] overhead, spent [658ms] collecting in the last [1.2s]
there are 3 master-eligible nodes, bellow is actual master node:
# top -n 1 | head
top - 06:06:42 up 5 days, 13:42, 2 users, load average: 0.31, 0.40, 0.35
Tasks: 93 total, 1 running, 92 sleeping, 0 stopped, 0 zombie
%Cpu(s): 5.2 us, 0.4 sy, 0.0 ni, 94.0 id, 0.0 wa, 0.0 hi, 0.4 si, 0.0 st
KiB Mem : 7663592 total, 4881024 free, 1861236 used, 921332 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 5512280 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
1376 alexus 20 0 3730832 1.479g 27460 S 13.3 20.2 891:58.28 java
647 root 20 0 691908 116228 43416 S 6.7 1.5 40:21.04 dockerd
1 root 20 0 56964 6656 5164 S 0.0 0.1 0:02.92 systemd
#
I used ES_HEAP_SIZE=4g
for elasticsearch (master-eligible nodes).
Please advise.