We are in the process of setting up Elasticsearch as a Search Engine to consume customer data. The Elasticsearch nodes are across 5 VMs in 2 different Datacenters (3:2), and all of them are Master-Data and active nodes. The index creation, data ingestion, data retrieval will be via Elastic APIs. No other Elasticstack components will be used.
Our problem is deciding on how to set this up when there is a complete Data center failure. We need clarity on how to avoid split brain condition, and ensure the cluster availability in such situations like complete Data center goes down.
We are providing the solution as open-source and cannot use the Cross Cluster Replication as mentioned in this blog.
How can we set this cluster up without any of the above mentioned scenarios?