Actually I have a cluster with 4 master nodes and 10 data nodes on a Kubernetes Cluster. Basically, each node causes my cluster to scale new hosts because of the anti-affinity default behavior, so my k8s got 14 nodes too. I was thinking about how to downsize my Elasticsearch cluster (and consequently my k8s cluster) and a idea I had was to deploy the stack with 3 (or even 2) master nodes and promote the 10 data nodes to non-eligible master, in addiction to their other roles. I guess that many voting nodes could prevent a split brain scenario, because at any time, even with no communication between 2 master nodes, the other nodes would vote and get into a consensus about which one is the master.
Is that a advisable step or shouldn't I do it?
I saw this related question but the asker was talking about turning all data nodes into eligible master nodes, which I know is not the best idea because of the risk of a highly demanded data node being elected as the cluster master and not having enough resources to act as a master.