Number of Master nodes across two data centers

Elastic Stack Elasticsearch
1

We are working on redesigning our currently cluster. Our 40 ES data nodes are split between 2 data centers, 20 in each. Currently there are 3 master dedicated nodes, 2 in one DC, and 1 in the other. The goal with the two datacenters would be to have the primary shard in one DC and the replica in the other.

Theoretically, if 1 datacenter was to go offline, a new master election would fail because there wouldn't be enough nodes for a quorum. If we increased the number of masters this problem continues as 3,5,7 etc. all will have the same issue.

Is there a solution to this problem?

Thanks!

2

You can not create a symmetrically highly available cluster across just 2 data centres. You therefore need a dedicated master node in a third data center to act as an arbiter. Voting only nodes have recently been introduced and could be used as arbiter but if you are on an earlier version a dedicated master node will be your best bet.

3

Ok, sounds like a master node in a third location is the best bet for use case.

Thanks for the input!