Elasticsearch 4 node cluster distributed in 2 rooms

[lecyberian (Le)]

Hey everyone,

I have a scenario where the requirement is having total 4 node cluster of elasticsearch having 2 in Site A and 2 in Site B.

Both sites are basically two floors connected with fiber, so network latency is not an issue.

The requirement is to have fault tolerance and high availability. For example: if Site A goes down Site B remains functional and vice versa.

I wanted to be sure to get expert advice on this before i implement, there is a parameter called "discovery.zen.minimum_master_nodes" if i set that to 3 in this case i suppose it means it needs 3 master servers to be available to form a quorum just to make sure its not a split brain in case of site goes down, However if Site A goes down then this requirement will never meet and quorum will not be form i would assume that Cluster would not be functional. Any suggestions here please?

Also what primary and replica shard settings do you suggest ? provided that there are 5 indexes of size each "200-250GB" and growing

Thanks!

[warkolm (Mark Walkom)]

This sort of situation is where you need to understand the tradeoffs you need to make.

1. Set it to 3 and then be prepared to restart the two nodes in the room that didn't go down to update the min master setting.
2. Have a tie-breaker node somewhere else.
3. Have the ability to spin up a 3rd node in either room to fulfil the 3 min.