# Elasticsearch master node replacment **URL:** \ **Category:** Elasticsearch\ **Created:** [July 23, 2019, 7:29pm UTC](https://discuss.elastic.co/t/elasticsearch-master-node-replacment/191901 "2019-07-23T19:29:00Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![ali\_tavakoli](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ali_tavakoli/32/50767_2.png) [@ali\_tavakoli](https://discuss.elastic.co/u/ali_tavakoli)\ **Post date:** [July 23, 2019, 7:29pm UTC](https://discuss.elastic.co/t/elasticsearch-master-node-replacment/191901/1 "2019-07-23T19:29:00Z")
Hi guys I run elasticsearch on 8 nodes with 2 different rack-id for shard replication awareness. i have 4 nodes on rack-1 and other 4 nodes on rack-2 and i have 1 master eligible node on each rack. so i have 2 master nodes and 6 data nodes. it's ok when i start all 8 node and master node elect and every thing is fine! when i stop master node i expect that second master eligible elect as master and cluster works fine but i have this error: " master not discovered or elected yet, an election requires a ..." Here is my config in master A: cluster.name: S-cluster node.name: node-a-1 node.attr.rack\_id: rack\_one path.data: /var/lib/elasticsearch path.logs: /var/log/elasticsearch bootstrap.memory\_lock: true network.host: _site_ discovery.seed\_hosts: ["192.168.100.20", "192.168.100.21", "192.168.100.22", "192.168.100.23", "192.168.100.25", "192.168.100.26", "192.168.100.27", "192.168.100.28"] cluster.initial\_master\_nodes: ["192.168.100.20", "192.168.100.25"] node.master: true node.data: true cluster.routing.allocation.awareness.attributes: rack\_id and here is my data node config: cluster.name: S-cluster node.name: node-a-2 node.attr.rack\_id: rack\_one path.data: /var/lib/elasticsearch path.logs: /var/log/elasticsearch bootstrap.memory\_lock: true network.host: 192.168.100.21 discovery.seed\_hosts: ["192.168.100.20", "192.168.100.21", "192.168.100.22", "192.168.100.23", "192.168.100.25", "192.168.100.26", "192.168.100.27", "192.168.100.28"] cluster.initial\_master\_nodes: ["192.168.100.20", "192.168.100.25"] node.master: false node.data: true and same config for master B and other data node! master IP is 192.168.100.20 and 192.168.100.25. Thanks for your help. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 23, 2019, 7:36pm UTC](https://discuss.elastic.co/t/elasticsearch-master-node-replacment/191901/2 "2019-07-23T19:36:08Z")
That is expected and the correct behaviour. Elasticsearch master election is based on consensus, so a majority of master eligible nodes are required in order to elect a master. The majority of 2 is 2, which is why a master can not be elected when 1 master eligible node goes missing. In order to be able to handle one master eligible node going down you therefore need at least 3 master eligible nodes in the cluster. ---
**Author:** ![ali\_tavakoli](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ali_tavakoli/32/50767_2.png) [@ali\_tavakoli](https://discuss.elastic.co/u/ali_tavakoli)\ **Post date:** [July 23, 2019, 7:39pm UTC](https://discuss.elastic.co/t/elasticsearch-master-node-replacment/191901/3 "2019-07-23T19:39:16Z")
Thanks for your quick reply. so what do u suggest? i run this cluster for fail over and fault tolerance. I want cluster works if one of my rack goes down! is this OK if i config 2 master eligible nodes on each rack? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 23, 2019, 7:44pm UTC](https://discuss.elastic.co/t/elasticsearch-master-node-replacment/191901/4 "2019-07-23T19:44:16Z")
That won’t help as the majority of 4 is 3. You generally need another node outside these racks. ---
**Author:** ![ali\_tavakoli](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ali_tavakoli/32/50767_2.png) [@ali\_tavakoli](https://discuss.elastic.co/u/ali_tavakoli)\ **Post date:** [July 23, 2019, 7:49pm UTC](https://discuss.elastic.co/t/elasticsearch-master-node-replacment/191901/5 "2019-07-23T19:49:06Z")
Can i change config manually when one of masters fail? is this OK if i change master eligible to healthy master node manually in all nodes and restart cluster? so i have a delay time to start cluster with all data. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 23, 2019, 7:57pm UTC](https://discuss.elastic.co/t/elasticsearch-master-node-replacment/191901/6 "2019-07-23T19:57:43Z")
If you have 4 master eligible nodes you can handle a single node going down, but not a full rack. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 23, 2019, 10:37pm UTC](https://discuss.elastic.co/t/elasticsearch-master-node-replacment/191901/7 "2019-07-23T22:37:08Z")
@Christian_Dahlqvist is absolutely correct, but to explain the issue a bit more: rack A cannot tell the difference between being disconnected from rack B and rack B actually failing; similarly rack B cannot tell the difference between being disconnected from rack A and rack A actually failing. Thus if the two racks were disconnected from each other and each of them believed that this meant the other rack had failed then both racks would elect a master and continue independently. This is called a _split-brain_ and results in data loss. Elasticsearch will not let this happen. It is impossible to solve this using only two racks. That's not a constraint that Elasticsearch imposes, it's a fundamental property of distributed systems. You need at least one more master-eligible node, independent of these two racks, to act as a tie-breaker in the case where the two racks are disconnected from each other. You should not attempt to override this manually, because this too can result in data loss. Adding an independent node is the only safe way to achieve what you ask. ---
**Author:** ![ali\_tavakoli](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ali_tavakoli/32/50767_2.png) [@ali\_tavakoli](https://discuss.elastic.co/u/ali_tavakoli)\ **Post date:** [July 24, 2019, 3:42am UTC](https://discuss.elastic.co/t/elasticsearch-master-node-replacment/191901/8 "2019-07-24T03:42:36Z")
Thanks a you so much guys, thats why i love elastic; becasue of you and your support. So i add another only master eligible node to cluster outside the racks, and i think it dosent need resourse as other nodes because it's just master node and not data node. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [July 24, 2019, 5:37am UTC](https://discuss.elastic.co/t/elasticsearch-master-node-replacment/191901/9 "2019-07-24T05:37:44Z")
Yes that's right, it won't need as much disk space if it's not a data node. When 7.3 is released you can also make it a [voting-only master-eligible node](https://www.elastic.co/guide/en/elasticsearch/reference/7.3/modules-node.html#voting-only-node) to prevent it from becoming master, which reduces its resource requirements further too. ---
**Author:** ![ali\_tavakoli](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ali_tavakoli/32/50767_2.png) [@ali\_tavakoli](https://discuss.elastic.co/u/ali_tavakoli)\ **Post date:** [July 24, 2019, 7:15am UTC](https://discuss.elastic.co/t/elasticsearch-master-node-replacment/191901/10 "2019-07-24T07:15:44Z")
Very nice thank you so much guys. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 21, 2019, 7:15am UTC](https://discuss.elastic.co/t/elasticsearch-master-node-replacment/191901/11 "2019-08-21T07:15:45Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.