Can I set multiple nodes as dedicated master nodes? Suppose I set 3 nodes as dedicated master nodes, and set minimum master nodes as 2, what benefit will I have as compared to having only a single dedicated master node?
That is the recommended way to do it. The benefit of having 3 dedicated master nodes is that you will be able to take down or lose one node without the cluster suffering, as the remaining master eligible node will be able to elect a new master to manage the cluster. This increases stability and availability and makes the cluster easier to upgrade through rolling upgrades without downtime.
That is the thing. Suppose I have 3 dedicated master nodes and minimum number of eligible master nodes is set to one. If one of my master node loses contact with the other 2 eligible master nodes, the remaining 2 nodes will re-elect a master nodes. There are now 2 master nodes available simultaneously. But, they are talking to the same set of data nodes. How did this affect my performance?
minimum_master_nodes should always be set to the number required to reach majority, e.g. 2 when using 3 dedicated master nodes, in order to avoid split brain scenarios.
In split brain scenario, more than one master node is elected right? But all of them are pointing to the same set of data nodes. So, what will be the issue???
If you set minimum_master_nodes to 2with 3 available master eligible nodes, only one master can be elected as 2 need to agree for that to happen. If a single node is partitioned off it will not be able to reach majority and it can therefore not elect itself master, meaning that will always be at most one (there may be none for periods while election takes place) elected master node in the cluster. There is therefore no risk of a split-brain scenario.