# Restart cluster ES2.1 failed after disk full exception, how can I make it?

**URL:** <https://discuss.elastic.co/t/restart-cluster-es2-1-failed-after-disk-full-exception-how-can-i-make-it/104514>\
**Category:** Elasticsearch\
**Created:** [October 19, 2017, 7:56am UTC](https://discuss.elastic.co/t/restart-cluster-es2-1-failed-after-disk-full-exception-how-can-i-make-it/104514 "2017-10-19T07:56:22Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![small-tomorrow](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/small-tomorrow/32/13319_2.png) [@small-tomorrow](https://discuss.elastic.co/u/small-tomorrow)\
**Post date:** [October 19, 2017, 7:56am UTC](https://discuss.elastic.co/t/restart-cluster-es2-1-failed-after-disk-full-exception-how-can-i-make-it/104514/1 "2017-10-19T07:56:22Z")

</div>

When I found the cluster was down, I found most nodes's disk is 100%, so I free most space by removing log files. but the cluster wont recovery and the Linux ES process seems blocked. So , I restart all nodes , and set one of the node as master by using node.master=true and others false.

**However, there're almost 60,000 shards , and the recovery will blocked when the active shards reach 60% more or less .**

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/c/6c553c2f0d73a642443f2c38cf4428a7cfa0963c.png)  
At that time , when I check the master stat use`jstat -gcutil`, it seems the JVM GC blocked .  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/1/d182063f82d4a379b808cbe143f1e748a46be35d.jpg)

**This is the first time that the whole cluster are down, and after hours I still can't restart it successfully ! So I beg you for help ! any help will be very grateful , waiting online**

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 16, 2017, 7:56am UTC](https://discuss.elastic.co/t/restart-cluster-es2-1-failed-after-disk-full-exception-how-can-i-make-it/104514/2 "2017-11-16T07:56:25Z")

</div>

Closed on November 16, 2017, 7:56am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.