How many ES nodes for 2 hardware servers

[avdelst (Andre Van Der Elst)]

Hi all,

When I have to server, each running multiple VM's what would be the correct number of ES nodes ?

The goal is to still have a functoning ES cluster even when 1 server completely dies due to a hardware failure.

Currently I have the following setup but something tells me ES would not survive a sudden los of one of the hosts:

Server A: ES01 , ES02
Server B: ES03, ES04

ES01,2,3,4 form a ES cluster where most settings are default.

Should I configure settings for number of replicas so that there's always a complete replica of the data on both of the hosts ?

Or is the best option to add another server and have 1 ES node per server ?

[Christian_Dahlqvist]

If you want the cluster to be able to continue operating fully when one full server is lost you will need a third server, as you will need 3 master eligible nodes on different hosts.

[mvg (Martijn Van Groningen)]

I think you should at least configure allocation awareness [1], in order to prevent both primary and replica shards to be allocated to the same physical server.

Whether you should have a single node per physical machine depends on how much jvm heap you configure per node. I would say if you need more than 26GB then you should have more than a single node [2], but this depends. Ideally you shouldn't need that much heap, as since ES 5 many data structures are not kept in the jvm heap, but in the OS' filesystem cache.

It would also be better, if you add a third physical server to your cluster, so you can set the minimum_master_nodes setting to avoid split brains [3].

1: https://www.elastic.co/guide/en/elasticsearch/reference/5.0/allocation-awareness.html
2: https://www.elastic.co/blog/a-heap-of-trouble
3: https://www.elastic.co/guide/en/elasticsearch/reference/5.0/modules-node.html#split-brain

[avdelst (Andre Van Der Elst)]

thanks everyone for all the replies.
I'll look into a 3rd server and I'll also have a look at allocation-awareness if i can't get a 3rd server.