Hi! Running 12 node Elasticsearch 2.2 cluster on Windows 2012, each node holds 10 shards (5 primary and 5 replica), shard size approx 15GB. I started experiencing slow garbage collection, like this:
[2016-06-10 07:46:50,250][WARN ][monitor.jvm ] [ESESSMW4061] [gc][young][1973921][15980] duration [18.2s], collections [1]/[18.4s], total [18.2s]/[1.2h], memory [9.4gb]->[6gb]/[15.4gb], all_pools {[young] [4.1gb]->[36.3mb]/[4.2gb]}{[survivor] [45.6mb]->[546.1mb]/[546.1mb]}{[old] [5.2gb]->[5.4gb]/[10.6gb]}
[2016-06-10 07:48:22,513][WARN ][monitor.jvm ] [ESESSMW4061] [gc][young][1973924][15981] duration [1.4m], collections [1]/[1.5m], total [1.4m]/[1.2h], memory [9.4gb]->[7.8gb]/[15.4gb], all_pools {[young] [3.4gb]->[3.3mb]/[4.2gb]}{[survivor] [546.1mb]->[546.1mb]/[546.1mb]}{[old] [5.4gb]->[7.3gb]/[10.6gb]}
[2016-06-10 07:48:52,614][WARN ][monitor.jvm ] [ESESSMW4061] [gc][young][1973937][15982] duration [17.8s], collections [1]/[18s], total [17.8s]/[1.2h], memory [12gb]->[8.5gb]/[15.4gb], all_pools {[young] [4.1gb]->[25.5mb]/[4.2gb]}{[survivor] [546.1mb]->[546.1mb]/[546.1mb]}{[old] [7.3gb]->[8gb]/[10.6gb]}
[2016-06-10 07:49:47,649][WARN ][monitor.jvm ] [ESESSMW4061] [gc][young][1973941][15983] duration [51.5s], collections [1]/[52s], total [51.5s]/[1.2h], memory [11.9gb]->[10.3gb]/[15.4gb], all_pools {[young] [3.4gb]->[20.5mb]/[4.2gb]}{[survivor] [546.1mb]->[546.1mb]/[546.1mb]}{[old] [8gb]->[9.8gb]/[10.6gb]}
[2016-06-10 07:54:05,394][WARN ][monitor.jvm ] [ESESSMW4061] [gc][old][1973947][12] duration [4.2m], collections [1]/[4.2m], total [4.2m]/[27.3m], memory [14.5gb]->[7gb]/[15.4gb], all_pools {[young] [4.1gb]->[7.4mb]/[4.2gb]}{[survivor] [546.1mb]->[0b]/[546.1mb]}{[old] [9.8gb]->[7gb]/[10.6gb]}
The node then disconnects form the cluster and connects again after a while.
I suspect that this is due to slow at times CPU (these are virtual nodes, 8 core 2.6ghz intel Xeon, 32 GB RAM/16GB to Elastic). Is that making sense and generally what can I do to resolve this? Thanks for your suggestions!