Hi, I am seeing a lot of gc overhead spent messages in the logs of a single node ES, there's also a log with total [858ms]/[21.7m]. How do I read these GCMonitor logs? What does this total [858ms]/[21.7m] means ?, What could be the cause of this?
We have about ~200 indices each with 1 shard, and total heap given to ES is 9 GB
Logs
INFO | jvm 1 | 2020/03/02 07:30:39 | [2020-03-02T07:30:39,456][WARN ][o.e.m.j.JvmGcMonitorService] [es_master_essvr16] [gc][219234] overhead, spent [664ms] collecting in the last [1s]
INFO | jvm 1 | 2020/03/02 07:30:40 | [2020-03-02T07:30:40,472][INFO ][o.e.m.j.JvmGcMonitorService] [es_master_essvr16] [gc][219235] overhead, spent [410ms] collecting in the last [1s]
INFO | jvm 1 | 2020/03/02 07:30:41 | [2020-03-02T07:30:41,644][INFO ][o.e.m.j.JvmGcMonitorService] [es_master_essvr16] [gc][219236] overhead, spent [583ms] collecting in the last [1.1s]
INFO | jvm 1 | 2020/03/02 07:30:42 | [2020-03-02T07:30:42,659][INFO ][o.e.m.j.JvmGcMonitorService] [es_master_essvr16] [gc][219237] overhead, spent [321ms] collecting in the last [1s]
INFO | jvm 1 | 2020/03/02 07:30:43 | [2020-03-02T07:30:43,831][INFO ][o.e.m.j.JvmGcMonitorService] [es_master_essvr16] [gc][young][219238][10908] duration [858ms], collections [1]/[1.1s], total [858ms]/[21.7m], memory [4.7gb]->[4.4gb]/[7.7gb], all_pools {[young] [355.3mb]->[17.1mb]/[865.3mb]}{[survivor] [10.3mb]->[12.2mb]/[108.1mb]}{[old] [4.4gb]->[4.4gb]/[6.7gb]}
This is happening continously!
INFO | jvm 1 | 2020/03/02 17:10:26 | [2020-03-02T17:10:26,023][INFO ][o.e.m.j.JvmGcMonitorService] [es_master_essvr16] [gc][253338] overhead, spent [264ms] collecting in the last [1s]
INFO | jvm 1 | 2020/03/02 17:11:23 | [2020-03-02T17:11:23,244][INFO ][o.e.m.j.JvmGcMonitorService] [es_master_essvr16] [gc][253392] overhead, spent [488ms] collecting in the last [1s]
INFO | jvm 1 | 2020/03/02 17:11:36 | [2020-03-02T17:11:36,448][WARN ][o.e.m.j.JvmGcMonitorService] [es_master_essvr16] [gc][253405] overhead, spent [530ms] collecting in the last [1s]
INFO | jvm 1 | 2020/03/02 17:11:52 | [2020-03-02T17:11:52,683][WARN ][o.e.m.j.JvmGcMonitorService] [es_master_essvr16] [gc][young][253420][12474] duration [1.2s], collections [1]/[2s], total [1.2s]/[28.4m], memory [4.3gb]->[3.6gb]/[7.7gb], all_pools {[young] [732.8mb]->[2.5mb]/[865.3mb]}{[survivor] [83.9mb]->[89.8mb]/[108.1mb]}{[old] [3.5gb]->[3.5gb]/[6.7gb]}
INFO | jvm 1 | 2020/03/02 17:11:52 | [2020-03-02T17:11:52,683][WARN ][o.e.m.j.JvmGcMonitorService] [es_master_essvr16] [gc][253420] overhead, spent [1.2s] collecting in the last [2s]
INFO | jvm 1 | 2020/03/02 17:12:21 | [2020-03-02T17:12:20,996][WARN ][o.e.m.j.JvmGcMonitorService] [es_master_essvr16] [gc][young][253447][12476] duration [1.4s], collections [1]/[1.9s], total [1.4s]/[28.5m], memory [4.3gb]->[3.6gb]/[7.7gb], all_pools {[young] [709.3mb]->[2mb]/[865.3mb]}{[survivor] [69.3mb]->[108.1mb]/[108.1mb]}{[old] [3.5gb]->[3.5gb]/[6.7gb]}
INFO | jvm 1 | 2020/03/02 17:12:21 | [2020-03-02T17:12:20,996][WARN ][o.e.m.j.JvmGcMonitorService] [es_master_essvr16] [gc][253447] overhead, spent [1.4s] collecting in the last [1.9s]
INFO | jvm 1 | 2020/03/02 17:12:33 | [2020-03-02T17:12:33,169][INFO ][o.e.m.j.JvmGcMonitorService] [es_master_essvr16] [gc][253459] overhead, spent [328ms] collecting in the last [1s]
INFO | jvm 1 | 2020/03/02 17:13:03 | [2020-03-02T17:13:03,076][INFO ][o.e.m.j.JvmGcMonitorService] [es_master_essvr16] [gc][253488] overhead, spent [551ms] collecting in the last [1.5s]
INFO | jvm 1 | 2020/03/02 17:19:06 | [2020-03-02T17:19:06,568][INFO ][o.e.m.j.JvmGcMonitorService] [es_master_essvr16] [gc][253846] overhead, spent [433ms] collecting in the last [1.2s]
INFO | jvm 1 | 2020/03/02 17:19:20 | [2020-03-02T17:19:20,553][WARN ][o.e.m.j.JvmGcMonitorService] [es_master_essvr16] [gc][young][253859][12492] duration [1.7s], collections [1]/[1.8s], total [1.7s]/[28.5m], memory [4.4gb]->[3.6gb]/[7.7gb], all_pools {[young] [860.7mb]->[28mb]/[865.3mb]}{[survivor] [74.3mb]->[75.3mb]/[108.1mb]}{[old] [3.5gb]->[3.5gb]/[6.7gb]}
INFO | jvm 1 | 2020/03/02 17:19:20 | [2020-03-02T17:19:20,553][WARN ][o.e.m.j.JvmGcMonitorService] [es_master_essvr16] [gc][253859] overhead, spent [1.7s] collecting in the last [1.8s]
INFO | jvm 1 | 2020/03/02 17:21:41 | [2020-03-02T17:21:41,795][INFO ][o.e.m.j.JvmGcMonitorService] [es_master_essvr16] [gc][253997] overhead, spent [311ms] collecting in the last [1s]
INFO | jvm 1 | 2020/03/02 17:22:41 | [2020-03-02T17:22:41,595][WARN ][o.e.m.j.JvmGcMonitorService] [es_master_essvr16] [gc][254056] overhead, spent [600ms] collecting in the last [1s]
INFO | jvm 1 | 2020/03/02 17:24:11 | [2020-03-02T17:24:11,835][INFO ][o.e.m.j.JvmGcMonitorService] [es_master_essvr16] [gc][254145] overhead, spent [352ms] collecting in the last [1s]
INFO | jvm 1 | 2020/03/02 17:25:33 | [2020-03-02T17:25:33,603][WARN ][o.e.m.j.JvmGcMonitorService] [es_master_essvr16] [gc][young][254224][12503] duration [1.8s], collections [1]/[2.6s], total [1.8s]/[28.6m], memory [4.3gb]->[3.5gb]/[7.7gb], all_pools {[young] [817.8mb]->[6.3mb]/[865.3mb]}{[survivor] [37.9mb]->[25.9mb]/[108.1mb]}{[old] [3.5gb]->[3.5gb]/[6.7gb]}
INFO | jvm 1 | 2020/03/02 17:25:33 | [2020-03-02T17:25:33,603][WARN ][o.e.m.j.JvmGcMonitorService] [es_master_essvr16] [gc][254224] overhead, spent [1.8s] collecting in the last [2.6s]
INFO | jvm 1 | 2020/03/02 17:26:57 | [2020-03-02T17:26:57,935][WARN ][o.e.m.j.JvmGcMonitorService] [es_master_essvr16] [gc][young][254304][12508] duration [4s], collections [1]/[4.1s], total [4s]/[28.7m], memory [4.4gb]->[3.5gb]/[7.7gb], all_pools {[young] [861.8mb]->[15.2mb]/[865.3mb]}{[survivor] [22.2mb]->[5.2mb]/[108.1mb]}{[old] [3.5gb]->[3.5gb]/[6.7gb]}
INFO | jvm 1 | 2020/03/02 17:26:57 | [2020-03-02T17:26:57,935][WARN ][o.e.m.j.JvmGcMonitorService] [es_master_essvr16] [gc][254304] overhead, spent [4s] collecting in the last [4.1s]
INFO | jvm 1 | 2020/03/02 17:27:13 | [2020-03-02T17:27:13,435][WARN ][o.e.m.j.JvmGcMonitorService] [es_master_essvr16] [gc][254319] overhead, spent [646ms] collecting in the last [1s]
INFO | jvm 1 | 2020/03/02 17:29:27 | [2020-03-02T17:29:27,551][WARN ][o.e.m.j.JvmGcMonitorService] [es_master_essvr16] [gc][young][254450][12523] duration [2.2s], collections [1]/[2.3s], total [2.2s]/[28.7m], memory [4.4gb]->[3.6gb]/[7.7gb], all_pools {[young] [825.5mb]->[25.1mb]/[865.3mb]}{[survivor] [40.7mb]->[86.5mb]/[108.1mb]}{[old] [3.5gb]->[3.5gb]/[6.7gb]}
INFO | jvm 1 | 2020/03/02 17:29:27 | [2020-03-02T17:29:27,551][WARN ][o.e.m.j.JvmGcMonitorService] [es_master_essvr16] [gc][254450] overhead, spent [2.2s] collecting in the last [2.3s]
INFO | jvm 1 | 2020/03/02 17:29:36 | [2020-03-02T17:29:36,302][WARN ][o.e.m.j.JvmGcMonitorService] [es_master_essvr16] [gc][young][254458][12524] duration [1.6s], collections [1]/[1.6s], total [1.6s]/[28.8m], memory [4.5gb]->[3.5gb]/[7.7gb], all_pools {[young] [865.3mb]->[9.6mb]/[865.3mb]}{[survivor] [86.5mb]->[7.4mb]/[108.1mb]}{[old] [3.5gb]->[3.5gb]/[6.7gb]}
INFO | jvm 1 | 2020/03/02 17:29:36 | [2020-03-02T17:29:36