# Catch-22 with minimum\_master\_nodes setting **URL:** \ **Category:** Elasticsearch\ **Created:** [October 1, 2015, 1:25pm UTC](https://discuss.elastic.co/t/catch-22-with-minimum-master-nodes-setting/31478 "2015-10-01T13:25:13Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![vhj](https://avatars.discourse-cdn.com/v4/letter/v/f14d63/32.png) [@vhj](https://discuss.elastic.co/u/vhj)\ **Post date:** [October 1, 2015, 1:25pm UTC](https://discuss.elastic.co/t/catch-22-with-minimum-master-nodes-setting/31478/1 "2015-10-01T13:25:13Z")
In my three node cluster I have minimum\_master\_nodes set to 2. Occasionally I need to drop down to 1 node for maintenance purposes. If I forget to set the minimum back to 1 before doing this then I end up stuck. Attempts to set minimum\_master\_nodes to 1 after dropping down to 1 node result in: curl -XPUT [http://127.0.0.1:9500/\_cluster/settings](http://127.0.0.1:9500/_cluster/settings) -d '{ "persistent" : { "discovery.zen.minimum\_master\_nodes" : 1} }' {"error":"MasterNotDiscoveredException[waited for [30s]]","status":503} Is there anyway to force this setting through? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [October 1, 2015, 1:39pm UTC](https://discuss.elastic.co/t/catch-22-with-minimum-master-nodes-setting/31478/2 "2015-10-01T13:39:41Z")
What if you change that setting _before_ you take down two of the three nodes? I don't get why you have to take two nodes offline at the same time. Unless you have two replicas the cluster would become red anyway. ---
**Author:** ![vhj](https://avatars.discourse-cdn.com/v4/letter/v/f14d63/32.png) [@vhj](https://discuss.elastic.co/u/vhj)\ **Post date:** [October 1, 2015, 1:58pm UTC](https://discuss.elastic.co/t/catch-22-with-minimum-master-nodes-setting/31478/3 "2015-10-01T13:58:53Z")
the reason why doesn't matter, the question is how to get out of this situation, it seems a bit poor that you can lock yourself out of the product like this if you forget to reduce min\_master\_nodes before reducing number of nodes. As I understand it the persistent settings take precedence over what's in the yml file, so you can't even set it in the yml and restart. ---
**Author:** ![vhj](https://avatars.discourse-cdn.com/v4/letter/v/f14d63/32.png) [@vhj](https://discuss.elastic.co/u/vhj)\ **Post date:** [October 2, 2015, 9:52am UTC](https://discuss.elastic.co/t/catch-22-with-minimum-master-nodes-setting/31478/4 "2015-10-02T09:52:11Z")
just an additional note, our customer isn't taking two nodes down at the same time, but one after the other - so the shards re-organize after the first node is removed and before the second is removed, then we end up with one node in a yellow state. It seems the only solution is to add back another node to cluster with it, but the customer may not always have another elasticsearch system available to do this. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [October 2, 2015, 11:33am UTC](https://discuss.elastic.co/t/catch-22-with-minimum-master-nodes-setting/31478/5 "2015-10-02T11:33:17Z")
> the reason why doesn't matter, the question is how to get out of this situation, it seems a bit poor that you can lock yourself out of the product like this if you forget to reduce min\_master\_nodes before reducing number of nodes. You can always spin up a temporary master-only instance to get out it. That instance can run on an existing machine. > just an additional note, our customer isn't taking two nodes down at the same time, but one after the other - so the shards re-organize after the first node is removed and before the second is removed, then we end up with one node in a yellow state. The customer should not to take down a node unless the cluster is green. ---
**Author:** ![vhj](https://avatars.discourse-cdn.com/v4/letter/v/f14d63/32.png) [@vhj](https://discuss.elastic.co/u/vhj)\ **Post date:** [October 2, 2015, 1:09pm UTC](https://discuss.elastic.co/t/catch-22-with-minimum-master-nodes-setting/31478/6 "2015-10-02T13:09:00Z")
Thanks Magnus, so spinning up a temporary system is the only way out of this. I can't reveal details of our product and how it works on public forums like this, but this is a very real scenario for us (it may sound daft, but there are valid reasons for it). I can put this advice on spinning up a temp system in our troubleshooting instructions. ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [October 2, 2015, 2:09pm UTC](https://discuss.elastic.co/t/catch-22-with-minimum-master-nodes-setting/31478/7 "2015-10-02T14:09:31Z")
You can easily work around that issue by slightly changing your cluster setup. The idea is to decouple master nodes from data nodes. Set up three master nodes (no client, no HTTP, no data - very small resource consumption) and set minimum master nodes to 3. Keep an eye on these nodes. They manage the cluster wide state and must always be up. Do not play with minimum master node setting. In case of having to remove a master node, add a new master node before removing the one you want to decommission. With the master nodes, you can add or remove client (HTTP) / data nodes as many as you wish, and in the order you wish. ---
**Author:** ![vhj](https://avatars.discourse-cdn.com/v4/letter/v/f14d63/32.png) [@vhj](https://discuss.elastic.co/u/vhj)\ **Post date:** [October 2, 2015, 2:18pm UTC](https://discuss.elastic.co/t/catch-22-with-minimum-master-nodes-setting/31478/8 "2015-10-02T14:18:52Z")
Thanks for taking the time to give the extra advice Jorg, unfortunately in the product I work on which embeds elasticsearch this isn't viable. We're actually breaking many of the ES recommendations, but it still works awesomely 99.999% of the time. I just needed to know if there was a way to 'break through' and reset min\_master\_nodes to 1 in the scenario I described. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 11:46pm UTC](https://discuss.elastic.co/t/catch-22-with-minimum-master-nodes-setting/31478/9 "2017-07-05T23:46:08Z")