Cluster Redundancy with VMware vSphere HA

I would like to use vSphere HA (or FT) to make my Elasticsearch cluster redundant, but I am wondering why I can't find a similar case study.
Does anyone have any information that can help me with this question?

The Elasticsearch cluster is configured as a virtual machine on an ESXi host, It has three master nodes and more than ten data nodes, with one replica There is.
It looks like it's already well resilient, but I'm thinking of getting a second ESXi host to add even more redundancy in case the ESXi host goes down.

In this case, I see that Elasticsearch recommends using inter-cluster replication and shard allocation.
high-availability-cluster-design-large-clusters

I've been looking for information other than www.elastic.co just to be sure. An example of using vSphere HA for Elasticsearch cluster redundancy is I didn't see it.
It's as if everyone is avoiding using vSphere HA for Elasticsearch redundancy.

I would appreciate if you could give me some information that would answer this question or give me a clue as to the answer.

Thank you for reading my question.

You have 13 nodes on one ESXi host? Impressive but you surely want to split them at least on 3 - but beyond that, this seems very generic. I'd think vSphere HA, etc. is for systems that are not inherently clustered or redundant. Elasticsearch is better than that :slight_smile:

That article is excellent, and as it mentions you should have a 3rd host holding a master, and separately, use shard allocation awareness which just tells Elasticsearch which nodes are on which host so it splits data across them (it will force the primary & replica on to nodes on separate machines):
https://www.elastic.co/guide/en/elasticsearch/reference/current/allocation-awareness.html

I know of some clusters that run in vSphere but it's not really a great idea IMO, as Steve says you don't really need the extra layer of HA functionality since Elasticsearch already does all that. 13 nodes on a single host is a lot; it's normally better to run fewer/larger nodes to avoid the per-node overhead, and/or to run them on more/smaller hosts to avoid a single-host failure from damaging too much at once.

I have seen cases where the two HA systems actually interfere with each other, particularly if you're thinking of relying on VMWare's "transparent" migrations of VMs between hosts. They're not always as transparent as claimed, and sometimes lead to Elasticsearch failing the migrating node.

1 Like

Hi Steve. Thanks for the reply.
I was planning to make it redundant with two ESXi hosts, but I will also consider a third host. The expected role of the third host is a tiebreaker so I can lower the specs.

Hi David. Thanks for the reply.
That's exactly what I wanted to know.
I was worried that even if vSphere HA (or FT) preserved the redundancy of the VMs, it might not preserve the redundancy of Elasticsearch running on top of VM.

your knowledge has been a great help to me in my project. Thank you for Steve and David support.

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.