we are facing a challenge with the redundancy of elasticsearch cluster. Our cluster consists of 2 VNFs where each VNF consists of 3 master elasticsearch VMs, 6 data elasticsearch VMs and 3 ingest elasticsearch VMs. One of the requirements we have is to have a VNF redundancy ensuring our cluster operates normally when 1 VNF goes down for any reason. Other requirement is to avoid a split brain situation in case of unexpected networking issues between the VNFs. So, in such elasticsearch cluster we would have 6 master nodes configured to be eligible for master election. To avoid split brain situation, for master election we configured that we need to have minimum of 4 master nodes available. But, this doesn't allow us to fulfill the second requirement of having elasticsearch cluster redundancy because if we lose the VNF, our remaining VNF doesn't have enough master nodes to fulfill the number of minimum available masters for master election.
What could a good practice in solving this issue? Having more elasticsearch master nodes eligible for master election?
All thoughts and ideas are welcome and appreciated.