I wanted to understand the behavior of split-brain with ElasticSearch.
So I created a two-node cluster with a single index. Once the cluster was established, I installed iptables rule like so:
iptables -I OUTPUT 1 -d node2 ip --jump DROP
iptables -I OUTPUT 1 -d node1 ip --jump DROP
As expected in some time I could see that node1 was a one-node cluster and node2 was an independent one-node cluster.
However, at this point when I delete the iptables entries, the cluster does NOT reform. I am using this with multicast.disabled on my on-prem servers.
My unicast.hosts file has on node1:
unicast.hosts: node2's IP
My unicast.hosts file has on node2:
unicast.hosts: node1's IP
So how can I get the cluster to re-form at this point? Is the only option a restart of elasticsearch service on one of the nodes??
Appreciate any responses. Thanks.