I have a scenario where the requirement is having total 4 node cluster of elasticsearch having 2 in Site A and 2 in Site B.
Both sites are basically two floors connected with fiber, so network latency is not an issue.
The requirement is to have fault tolerance and high availability. For example: if Site A goes down Site B remains functional and vice versa.
I wanted to be sure to get expert advice on this before i implement, there is a parameter called "discovery.zen.minimum_master_nodes" if i set that to 3 in this case i suppose it means it needs 3 master servers to be available to form a quorum just to make sure its not a split brain in case of site goes down, However if Site A goes down then this requirement will never meet and quorum will not be form i would assume that Cluster would not be functional. Any suggestions here please?
Also what primary and replica shard settings do you suggest ? provided that there are 5 indexes of size each "200-250GB" and growing