Recommended number of master eligible nodes in a cluster


(Darshat Shah) #1

Hi,
What is the guideline on recommended number of master eligible nodes in a
cluster?

We have a big cluster with 90+ nodes, currently all are eligible to be
masters. However I do see master election take very long after a cluster
restart - most nodes are still trying to ping the old master and I think
there is no way to force a re-election.

Will re-election be faster if number of master eligible nodes be less? Is
there a rule of thumb, given N nodes how many should we configure to be
eligible to be masters?

Thanks
Darshat

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c573961b-29eb-45e8-a5f5-9b31175efb46%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(drjz) #2

The formula is N/2+1, so in your case that would be 46 if there are 90
nodes.

/JZ

On Wednesday, January 14, 2015 at 9:38:01 AM UTC+1, Darshat Shah wrote:

Hi,
What is the guideline on recommended number of master eligible nodes in a
cluster?

We have a big cluster with 90+ nodes, currently all are eligible to be
masters. However I do see master election take very long after a cluster
restart - most nodes are still trying to ping the old master and I think
there is no way to force a re-election.

Will re-election be faster if number of master eligible nodes be less? Is
there a rule of thumb, given N nodes how many should we configure to be
eligible to be masters?

Thanks
Darshat

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c2a88fd9-209b-4e7e-bbfe-03c8cb5e82a6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Darshat Shah) #3

That's the minimum number. Is it recommended to have more, or will
elections take longer if we have more than this minimum?

On Wednesday, January 14, 2015 at 2:16:27 PM UTC+5:30, drjz wrote:

The formula is N/2+1, so in your case that would be 46 if there are 90
nodes.

/JZ

On Wednesday, January 14, 2015 at 9:38:01 AM UTC+1, Darshat Shah wrote:

Hi,
What is the guideline on recommended number of master eligible nodes in a
cluster?

We have a big cluster with 90+ nodes, currently all are eligible to be
masters. However I do see master election take very long after a cluster
restart - most nodes are still trying to ping the old master and I think
there is no way to force a re-election.

Will re-election be faster if number of master eligible nodes be less? Is
there a rule of thumb, given N nodes how many should we configure to be
eligible to be masters?

Thanks
Darshat

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c73fdcf8-e5c1-42ef-82f3-0e0d4f897cf0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Darshat Shah) #4

To clarify, the guideline I'm looking for is how many nodes to set apart as
dedicated to be master eligible in a cluster of size N. That is not N/2+1.

On Wednesday, January 14, 2015 at 2:24:29 PM UTC+5:30, Darshat Shah wrote:

That's the minimum number. Is it recommended to have more, or will
elections take longer if we have more than this minimum?

On Wednesday, January 14, 2015 at 2:16:27 PM UTC+5:30, drjz wrote:

The formula is N/2+1, so in your case that would be 46 if there are 90
nodes.

/JZ

On Wednesday, January 14, 2015 at 9:38:01 AM UTC+1, Darshat Shah wrote:

Hi,
What is the guideline on recommended number of master eligible nodes in
a cluster?

We have a big cluster with 90+ nodes, currently all are eligible to be
masters. However I do see master election take very long after a cluster
restart - most nodes are still trying to ping the old master and I think
there is no way to force a re-election.

Will re-election be faster if number of master eligible nodes be less?
Is there a rule of thumb, given N nodes how many should we configure to be
eligible to be masters?

Thanks
Darshat

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e8e6dd50-203f-4ce7-9c78-3a989c526af6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Mark Harwood-2) #5

For installations like yours we would recommend 3 master eligible nodes.
This number doesn't need to scale with data volumes as your data nodes do.

A master eligible node should also be dedicated to that task alone - trying
to also fulfil search or indexing activities puts them at the risk of
becoming unresponsive.
We recommend 3 master-eligible nodes because that gives you both a
resiliency to failure and a necessary quorum required to avoid split-brain.
Fortunately master nodes can be lighter-weight machines than those
dedicated to search.

Anything less than 3 dedicated master-eligible nodes is essentially an
architectural compromise made for smaller-budget configurations.

On Wednesday, January 14, 2015 at 9:39:47 AM UTC, Darshat Shah wrote:

To clarify, the guideline I'm looking for is how many nodes to set apart
as dedicated to be master eligible in a cluster of size N. That is not
N/2+1.

On Wednesday, January 14, 2015 at 2:24:29 PM UTC+5:30, Darshat Shah wrote:

That's the minimum number. Is it recommended to have more, or will
elections take longer if we have more than this minimum?

On Wednesday, January 14, 2015 at 2:16:27 PM UTC+5:30, drjz wrote:

The formula is N/2+1, so in your case that would be 46 if there are 90
nodes.

/JZ

On Wednesday, January 14, 2015 at 9:38:01 AM UTC+1, Darshat Shah wrote:

Hi,
What is the guideline on recommended number of master eligible nodes in
a cluster?

We have a big cluster with 90+ nodes, currently all are eligible to be
masters. However I do see master election take very long after a cluster
restart - most nodes are still trying to ping the old master and I think
there is no way to force a re-election.

Will re-election be faster if number of master eligible nodes be less?
Is there a rule of thumb, given N nodes how many should we configure to be
eligible to be masters?

Thanks
Darshat

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0e1c31b3-7682-4009-bc79-b46a9c6cda2a%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Ed Kim) #6

3 should be fine, and they don't even have to be beefy machines.

On Wednesday, January 14, 2015 at 1:39:47 AM UTC-8, Darshat Shah wrote:

To clarify, the guideline I'm looking for is how many nodes to set apart
as dedicated to be master eligible in a cluster of size N. That is not
N/2+1.

On Wednesday, January 14, 2015 at 2:24:29 PM UTC+5:30, Darshat Shah wrote:

That's the minimum number. Is it recommended to have more, or will
elections take longer if we have more than this minimum?

On Wednesday, January 14, 2015 at 2:16:27 PM UTC+5:30, drjz wrote:

The formula is N/2+1, so in your case that would be 46 if there are 90
nodes.

/JZ

On Wednesday, January 14, 2015 at 9:38:01 AM UTC+1, Darshat Shah wrote:

Hi,
What is the guideline on recommended number of master eligible nodes in
a cluster?

We have a big cluster with 90+ nodes, currently all are eligible to be
masters. However I do see master election take very long after a cluster
restart - most nodes are still trying to ping the old master and I think
there is no way to force a re-election.

Will re-election be faster if number of master eligible nodes be less?
Is there a rule of thumb, given N nodes how many should we configure to be
eligible to be masters?

Thanks
Darshat

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/bcec37c0-e62e-4475-9042-2f955c2f918e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Matías Waisgold) #7

I´ve created a 200 nodes cluster and successed with 3 master_eligible
nodes. The important thing is that they have the data_node setting in
false, and data_nodes have it the other way around. This will improve a lot
your cluster resiliency.

On Wednesday, January 14, 2015 at 4:22:58 PM UTC-3, Ed Kim wrote:

3 should be fine, and they don't even have to be beefy machines.

On Wednesday, January 14, 2015 at 1:39:47 AM UTC-8, Darshat Shah wrote:

To clarify, the guideline I'm looking for is how many nodes to set apart
as dedicated to be master eligible in a cluster of size N. That is not
N/2+1.

On Wednesday, January 14, 2015 at 2:24:29 PM UTC+5:30, Darshat Shah wrote:

That's the minimum number. Is it recommended to have more, or will
elections take longer if we have more than this minimum?

On Wednesday, January 14, 2015 at 2:16:27 PM UTC+5:30, drjz wrote:

The formula is N/2+1, so in your case that would be 46 if there are 90
nodes.

/JZ

On Wednesday, January 14, 2015 at 9:38:01 AM UTC+1, Darshat Shah wrote:

Hi,
What is the guideline on recommended number of master eligible nodes
in a cluster?

We have a big cluster with 90+ nodes, currently all are eligible to be
masters. However I do see master election take very long after a cluster
restart - most nodes are still trying to ping the old master and I think
there is no way to force a re-election.

Will re-election be faster if number of master eligible nodes be less?
Is there a rule of thumb, given N nodes how many should we configure to be
eligible to be masters?

Thanks
Darshat

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6d952146-4f2f-4c94-952e-dcc852f0be57%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Xavier Facq) #8

Hi,

@Matias_Waisgold : With your cluster of 200 nodes with 3 master_eligble, does all nodes vote for the master election or only masters ? If only masters vote, what is the configuration of above parameters ?

discovery.zen.minimum_master_nodes: 2 ==> I think yes , doc says : (3 / 2 + 1)

discovery.zen.master_election.filter_data = true (only master vote )

Thank you !


(system) #9