We have a 2 nodes ES cluster.
Each node is in a separate datacenter.
There has been an issue with the network that connects the 2 datacenters and now we have a split brain issue.
To avoid the split brain problem in the future, we are thinking about adding a 3rd node in the cluster.
The sole purpose of the node would be to avoid the split brain problem, so we want use the minimum needed hardware resources for the 3rd node.
ES node 1 [already deployed in Datacenter 1]: 32 GB, 4 cores, node.master: true, node.data: true
ES node 2 [already deployed in Datacenter 2]: 32 GB, 4 cores, node.master: true, node.data: true
ES node 3 [about to be deployed in Datacenter 1]: 8 GB, 2 cores, node.master:true, node.data: false
Q1: Is the above setup with 3 ES nodes avoiding "split brain" issues?
Q2: Are there any problems with the above setup regarding the cluster High Availability?
i.e., Is everything fine when:
a. (ES node 1) fails or
b. (ES node 2) fails or
c. the network between the datacenters fails
Q3: What happens when both (ES node 1) and (ES node 2) are down and (ES node 3) is up? Are applications making use of the ES cluster receiving an error?