This is the exact setup and settings:
3 node cluster, node A is the original node, node B is added, followed by node C
Node A is already running before B starts, so I can automatically detect that B will be clustering with A before it actually starts. When B starts, I pass the discovery.zen.ping.unicast.hosts as a start up setting configured with A's IP.
Nodes A & B are already running before C starts, so I can automatically detect that C will be clustering with A & B before it actually starts. When C starts, I pass the discovery.zen.ping.unicast.hosts as a start up setting configured with IP's of A & B.
discovery.zen.minimum_master_nodes is set to 2
Once the cluster is established I periodically programatically update the yml file on each node, setting discovery.zen.ping.unicast.hosts with the IP's of all other nodes in the cluster. So the configuration in the yml file looks like this:
Node A: discovery.zen.ping.unicast.hosts: ["nodeB_IP", "nodeC_IP"]
Node B: discovery.zen.ping.unicast.hosts: ["nodeA_IP", "nodeC_IP"]
Node C: discovery.zen.ping.unicast.hosts: ["nodeA_IP", "nodeB_IP"]
I do this in case any of the nodes are restarted, but I do not intentionally restart any of the nodes at this point.
The whole idea is to prevent the split brain scenario from occurring and to automatically recover in the case that any node loses visibility of the cluster. So once the cluster is setup (having not restarted any of the nodes), each node's discovery.zen.ping.unicast.hosts list is only configured with the nodes that were visible when it started (since it was passed as start up setting).
If either B or C lose sight of the cluster, they are able to rejoin when visibility is established. They were started with A's IP in their discovery.zen.ping.unicast.hosts list.
However, if the other two nodes lose sight of A (say due to a network failure), they are able to form a cluster and elect a new master, but A is not able to rejoin the cluster after visibility is restored. Presumably this is because A does not have either B or C in its discovery.zen.ping.unicast.hosts yet (since it has not been restarted).
So what I would like to do is have a way for a A to re-join the cluster without needing to restart. Is this possible?