When I found the cluster was down, I found most nodes's disk is 100%, so I free most space by removing log files. but the cluster wont recovery and the Linux ES process seems blocked. So , I restart all nodes , and set one of the node as master by using node.master=true and others false.
However, there're almost 60,000 shards , and the recovery will blocked when the active shards reach 60% more or less .
At that time , when I check the master stat use
jstat -gcutil, it seems the JVM GC blocked .
This is the first time that the whole cluster are down, and after hours I still can't restart it successfully ! So I beg you for help ! any help will be very grateful , waiting online