Need more clarity and understanding on Split brain problem?

Elastic Stack Elasticsearch
1

Hi Experts,

I have 4 node cluster that is 2 servers and each server has 2 nodes . I have default config for all the four nodes that is
master node:true
data node:true

So my first query would be is there any possibility that I can face Split brain problem in this architecture(as per my understanding I will never because I have 4 nodes and anyone can be elected as master).
Second query would be is it necessary to have a dedicated master node , if yes then per cluster 1 master node i or per cluster 2 master node is the good selection ?

Thanks
VG

1 Like
2

The critical factor with respect to split-brain scenarios is not necessarily how many master eligible nodes you have but rather that you have set minimum_master_nodes to an appropriate value. As Elasticsearch is a consensus based system, it is important that a majority of master eligible nodes agree when a master node is elected. If this was not the case two parts of the cluster could end up electing different master nodes, causing a split-brain scenario.

In order to prevent this you should set minimum_master_nodes to FLOOR((2/N) + 1) assuming there is more than 1 master eligible node. If you have 4 master eligible nodes it should therefore be set to 3.

Having a single master eligible node eliminates the risk of split-brain scenarios, but also does not allow the cluster to continue operating properly if this node goes down.

3

thanks Chris,
so what is your suggestion on

Second query would be is it necessary to have a dedicated master node , if yes then per cluster 1 master node or per cluster 2 master node is the good selection ?

4

You can have all 4 nodes be master eligible and set minimum_master_nodes to 3. This will allow the cluster to continue operate even if you lose one node, e.g. through crash or maintenance.

2 Likes
5

Chris,

I have one node at the moment but I'm looking to add a second for both performance and and redundency benfits. In that instance how should I manage it? if I set the minimum master nodes to 2 then both need to be up so do I loose the redundency?

I know getting a 3rd node would be ideal but I don't think I can get the budget approval for that at the moment.

Many thanks for your help

6

If you have 2 nodes, minimum_master_nodes should be set to 2. This means that if one node goes down or the two nodes are partitioned off, you will be able to search but not index any more documents. If you need high write availability, you need 3 nodes, but that can be two master/data nodes and a small, dedicated master node.

7

I thought that might be the solution but wasn't sure of the implications, that helps a lot, thank you!

8

one more thing actually, you menation a small master node? do I take it from that I can introduce a node which is only there to manage indexes and doesn't hold any of the data shards itself? I ask as this would be a viable solution for me as I could depoloy a VM on our hyperV to do this role until we had the business need for a 3rd physical box in the cluster, I can't run a VM that takes the data shards though as it would put to much load on the shared NIC and disk pack.