Minimum master nodes in ES cluster

[Rupesh_Ranjan]

I have a ES cluster with nodes across data-centres with 1 coordinator node and 3 master+data nodes in each DC.

I have 2 questions here:

1. Is it any harm in using nodes which are acting as both master and data?
2. For minimum number of master nodes, as per the rule i should be mentioning 4 as min no. of nodes, but in that case, if in case something happens to one of the DCs, my cluster wont come back up. What is the suggested way to handle this scenario?

[Christian_Dahlqvist]

Unless the connection between the two data centres has very good throughput and very low latency, this type of deployment is not recommended as Elasticsearch requires good and reliable connections within a cluster.

@Rupesh_Ranjan

For minimum number of master nodes, as per the rule i should be mentioning 4 as min no. of nodes, but in that case, if in case something happens to one of the DCs, my cluster wont come back up. What is the suggested way to handle this scenario?

With just two data centres, you can not configure a symmetrically highly available cluster where Elasticsearch will continue operating when any of the data centres is lost. To get this you will need a third DC.

@Rupesh_Ranjan

Is it any harm in using nodes which are acting as both master and data?

This works for many scenarios, but can cause problems if the nodes come under high load, in which case dedicated master nodes that do not serve traffic is useful.

[Rupesh_Ranjan]

Thanks, christian.

Can you please elaborate more on

@Christian_Dahlqvist

dedicated master nodes that do not serve traffic

what kind of traffic are you mentioning here? Read or write? And is that by default for a dedicated master node?
Just to add, I am using coordinate node to call ES APIs wherever required.

[Christian_Dahlqvist]

Dedicated master nodes should ideally be left to just monitor the cluster. This means that they do not serve read or write traffic and can be much smaller than the data nodes.

[Rupesh_Ranjan]

Unfortunately, i dont have luxury of adding a third datacenter, so i'll have to live with 2 DCs. I understand the risk that either i can avoid split brain, or provide safety from datacenter failures.

So, this is what i am planning to keep
DC-1 : 1 coordinate, 1 master+data, 1 dedicated master, 2 dedicated data
DC-2 : 1 coordinate, 1 dedicated master, 3 dedicated data
min no. of master eligible nodes: 2
And, use coordinate nodes to communicate to cluster

With this config, is it too many a dedicated master nodes?

[Christian_Dahlqvist]

No that seems like a good number as you almost always want to have 3 master-eligible nodes.

[Rupesh_Ranjan]

Since, there can be only one master at a time, and lets say the node which is master+data, get elected as master. In that case, can i still have issues if there is load, because as you mentioned if the nodes come under high load, dedicated master nodes are helpful.

[Christian_Dahlqvist]

You could, but if you do it is likely one of the dedicated master nodes may get elected.

[Rupesh_Ranjan]

Terrific. Thanks a lot, it helps