Master node re-join cluster after network outage

Elastic Stack Elasticsearch
1

Cluster Setup:
I have a cluster with one original node (i.e. a node that does not have discovery.zen.ping.unicast.hosts configured). Multicast discovery is disabled and unicast is enabled. Other nodes are added to the cluster by configuring their discovery.zen.ping.unicast.hosts list with the IPs of the current nodes in the cluster. The discovery.zen.minimum_master_nodes setting is set to 2/N + 1 on each node, where N is the number of nodes in the cluster.

Scenario:
The network cable on the master node is unplugged.

Result:
The remaining nodes are able to elect a new master. When the network connection to the old master is restored, it cannot rejoin the cluster.

Question:
I would like to have the old master node be able to rejoin the cluster without needing to configure discovery.zen.ping.unicast.hosts in the yml file and restart. Is this possible?

Elasticsearch version 2.3.1

2

Why do you need to do this?
What hosts do you have listed in the setting?

3

This is the exact setup and settings:

  • 3 node cluster, node A is the original node, node B is added, followed by node C

  • Node A is already running before B starts, so I can automatically detect that B will be clustering with A before it actually starts. When B starts, I pass the discovery.zen.ping.unicast.hosts as a start up setting configured with A's IP.

  • Nodes A & B are already running before C starts, so I can automatically detect that C will be clustering with A & B before it actually starts. When C starts, I pass the discovery.zen.ping.unicast.hosts as a start up setting configured with IP's of A & B.

  • discovery.zen.minimum_master_nodes is set to 2

  • Once the cluster is established I periodically programatically update the yml file on each node, setting discovery.zen.ping.unicast.hosts with the IP's of all other nodes in the cluster. So the configuration in the yml file looks like this:
    Node A: discovery.zen.ping.unicast.hosts: ["nodeB_IP", "nodeC_IP"]
    Node B: discovery.zen.ping.unicast.hosts: ["nodeA_IP", "nodeC_IP"]
    Node C: discovery.zen.ping.unicast.hosts: ["nodeA_IP", "nodeB_IP"]
    I do this in case any of the nodes are restarted, but I do not intentionally restart any of the nodes at this point.

The whole idea is to prevent the split brain scenario from occurring and to automatically recover in the case that any node loses visibility of the cluster. So once the cluster is setup (having not restarted any of the nodes), each node's discovery.zen.ping.unicast.hosts list is only configured with the nodes that were visible when it started (since it was passed as start up setting).

If either B or C lose sight of the cluster, they are able to rejoin when visibility is established. They were started with A's IP in their discovery.zen.ping.unicast.hosts list.

However, if the other two nodes lose sight of A (say due to a network failure), they are able to form a cluster and elect a new master, but A is not able to rejoin the cluster after visibility is restored. Presumably this is because A does not have either B or C in its discovery.zen.ping.unicast.hosts yet (since it has not been restarted).

So what I would like to do is have a way for a A to re-join the cluster without needing to restart. Is this possible?

4

Why not set the unicast list to all 3 masters before starting/restarting?

5

Not all nodes are known when the original master starts up, so it cannot have other nodes in its unicast list when it starts up.