# Cluster locks up if master node filesystem becomes read-only

**URL:** <https://discuss.elastic.co/t/cluster-locks-up-if-master-node-filesystem-becomes-read-only/38245>\
**Category:** Elasticsearch\
**Created:** [January 1, 2016, 3:29pm UTC](https://discuss.elastic.co/t/cluster-locks-up-if-master-node-filesystem-becomes-read-only/38245 "2016-01-01T15:29:21Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Paul\_Maseberg](https://avatars.discourse-cdn.com/v4/letter/p/439d5e/32.png) [@Paul\_Maseberg](https://discuss.elastic.co/u/Paul_Maseberg)\
**Post date:** [January 1, 2016, 3:29pm UTC](https://discuss.elastic.co/t/cluster-locks-up-if-master-node-filesystem-becomes-read-only/38245/1 "2016-01-01T15:29:22Z")

</div>

Have a three node cluster running elasticsearch 2.1.0. The master node is also a data node. The machine that was master had hardware issues and the filesystem became read only. The whole cluster locked up until I killed the elasticsearch process on the bad machine.

Is there a setting or something that can be used to make the cluster drop a node that is still running elasticsearch but is having issues like a readonly filesystem?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [January 1, 2016, 11:51pm UTC](https://discuss.elastic.co/t/cluster-locks-up-if-master-node-filesystem-becomes-read-only/38245/2 "2016-01-01T23:51:01Z")

</div>

No there's not.

---

<div class="post-metadata">

**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\
**Post date:** [January 2, 2016, 12:51am UTC](https://discuss.elastic.co/t/cluster-locks-up-if-master-node-filesystem-becomes-read-only/38245/3 "2016-01-02T00:51:04Z")

</div>

There are two solutions.

First one is ES-only. The reason why ES is not shutting down automatically is because `org.elasticsearch.env.NodeEnviroment` keeps a `java.nio.file.FileStore` which is never monitored by calling `isReadOnly()` method regularly.

The `java.nio.file.FileStore` of all writable paths would have to be monitored for emergency stop in such kind of event.

To make the cluster drop a node with readonly file store, you would have modify the code and submit a patch.

The second solution: to detect general hardware malfunctions, JVM-based methods are quite not sufficient. Hence, ES is the not the best place to implement that, but the OS. You have to set up server monitoring software which can understand SNMP or IPMI or triggers for mcelog [https://github.com/andikleen/mcelog](https://github.com/andikleen/mcelog) that can kill ES (and other) processes in case of severe events.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 11:27pm UTC](https://discuss.elastic.co/t/cluster-locks-up-if-master-node-filesystem-becomes-read-only/38245/4 "2017-07-05T23:27:31Z")

</div>