3 master + data node setup

Elastic Stack Elasticsearch
1

This question may asked before, but I can't find a definitive answer here...

My boss currently ask me to design a ES cluster, but with budget constrain...

So a default 3 master-dedicated node and also 5 data-dedicated node is a no go.

Lastly my boss asked me:

"Why not create a cluster with 3 default ES roles nodes? (Means master + data + client etc) Just set minimum master nodes to 2, then no split brain! Why need dedicated master node to begin with?"

I don't even know how to retort to my boss since in my knowledge, the purpose of dedicated master node is to prevent split brain...

So here are the questions:

  1. Is there any consequences / possible issues arise if I go with the 3 default all-in-one roles nodes as a cluster?
  2. Why we need dedicated master nodes to begin with?

Appreciate on your helps!

2

I'd say that your boss is right here.
On small clusters, 3 nodes in total is enough.

And even more, if your data fit, you can also have:

2 "big" master+data nodes
1 small master only node (tie-breaker node with fewer memory)

That will reduce your bill even more.

3

The idea of dedicated masters is to protect your cluster as much as possible.

There's nothing wrong with using the default roles and having 3/5 nodes, you just run a higher risk of the active master being over run by queries or index load and then causing a re-election.

4

So it means if I set my minimum-master right, let's say in this case is 3 nodes, I should set the value to 2, so even if one of my nodes down, split brain won't happen, am I right?

5

Yes.