How to add more masters to a cluster of one master whilst avoiding split-brain and downtime?

Elastic Stack Elasticsearch
1

I have a three node cluster with 1 master-data node and 2 data-only nodes. I want to make this into a 3 master-data node cluster without any downtime. To this effect, I planned to do the following:

  1. First, set discovery.zen.minimum_master: 2 setting dynamically via the REST API.
  2. Second, set node.master and node.data to true in the config file of each data node and restart - one at a time.

However, this may lead to split-brain since the current discovery.zen.minimum_master setting default to 1, and I have been suggested to do a full cluster restart to avoid this. But since that will cause downtime, is there a way around this?

2

But if I do a rolling restart (after changing cluster quorum to 2 dynamically) whereby I change config on one data node (to make it master-data node) and restart, there could still be a split-brain since current master will self-elect while new master-eligible nodes would elect master too as they need 2 nodes to elect master. Hence, I have been suggested to do a full cluster restart to avoid this. But since cluster restart will cause downtime, is there a way around this?

Elasticsearch version is 1.5

3

No, there is at present no fully reliable way to change a cluster from having a single master node to having two master nodes for exactly the reasons you describe (although 2 to 3 works fine since no change to minimum_master_nodes is required). The only truly safe way to do this is with a full cluster restart.

1.5 hit end-of-life over two years ago, and has known resilience bugs. You should upgrade. Perhaps you could stand up a newer cluster alongside this one, use reindex-from-remote to copy the data across, and then cut across to your new cluster without downtime.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.