I have done some reading about the splitbrain scenario, but seems to have hit a dilemma.
e.g for this structure:
machine1: 2 ES nodes (with rack id 1)
machine2: 2 ES nodes (with rack id 2)
In the above case, all the settings are default, except for awareness, which i wanted the pri/replica shards to be distributed across racks.
So if there is a network issue between the 2 machines, I will have a split brain scenario since min. master is 1.
- I can set min. master to 3 to avoid split brain:
- but ES will be inoperable if network issue happens or even
- if 1 machine goes down, the ES is also inoperable, thus losing HA
- if I set min.master to anything less than 3,
- I will get split brain if there is network issue between the 2 machines
- but if 1 machine goes down, I can still have 1 machine which is working.(HA)
I am not sure this is a right design or I am missing some better configs.
Any advise will be great!