Master fall

Hi,

What's the best practice for an elasticsearch client to handle a scenario
in which elasticsearch master falls?

From my tests I can connect to any of the slaves and continue on. Does it
matter that the client connect to a slave and not to a master?

Thanks,
Ophir

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Elasticsearch will automatically elect a new master if the old one becomes
unavailable, but master's are not necessary for search/indexing operations.
E.g. any node in the cluster is fully capable of servicing a search
request, not just the master node.

The master is in charge of managing cluster metadata and state (mappings,
new indices, etc), in addition to normal indexing/search requirements.

It is also important to have your minimum_master_nodes setting configured
properly, to help avoid split-brains.

-Zach

On Tuesday, February 19, 2013 7:59:53 AM UTC-5, Ophir Michaeli wrote:

Hi,

What's the best practice for an elasticsearch client to handle a scenario
in which elasticsearch master falls?

From my tests I can connect to any of the slaves and continue on. Does it
matter that the client connect to a slave and not to a master?

Thanks,
Ophir

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hi,

It is not clear from the documentation why minimum_master_nodes should be
higher than 1 for more than 2 nodes, and what are the considerations to
configure properly.

Thanks

On Tuesday, February 19, 2013 3:17:14 PM UTC+2, Zachary Tong wrote:

Elasticsearch will automatically elect a new master if the old one becomes
unavailable, but master's are not necessary for search/indexing operations.
E.g. any node in the cluster is fully capable of servicing a search
request, not just the master node.

The master is in charge of managing cluster metadata and state (mappings,
new indices, etc), in addition to normal indexing/search requirements.

It is also important to have your minimum_master_nodes setting configured
properly, to help avoid split-brains.

-Zach

On Tuesday, February 19, 2013 7:59:53 AM UTC-5, Ophir Michaeli wrote:

Hi,

What's the best practice for an elasticsearch client to handle a scenario
in which elasticsearch master falls?

From my tests I can connect to any of the slaves and continue on. Does it
matter that the client connect to a slave and not to a master?

Thanks,
Ophir

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

The recommended value is usually (N/2 + 1) where N is the number of
master-eligible nodes.

For example, if you have min_master_nodes == 1 in a three node cluster, you
can easily get a "split-brain". Imagine one node drops out of the cluster,
but remains active (e.g. the server didn't die, but the network connection
between it and the rest of the nodes died). This solitary node sees the
other two departing and thinks it is the only survivor, so it elects itself
as master and forms a new cluster.

Meanwhile, the other two nodes see the single node leave and just carry on
as usual.

Now, when the network disconnect resolves itself, the third node does not
attempt to rejoin the cluster because it thinks it is the cluster and the
other two are missing. This is a split-brain, where you have two clusters
when only one should exist. Even worse, they don't realize there are two
clusters so now you have rapidly diverging data states because they aren't
trying to remerge.

Min_master_nodes prevents this. If min_master_nodes == 2 in the above
scenario, the solitary node sees only one master eligible node (itself),
which is less than the required 2 nodes. Therefore, instead of electing
itself as master, it simply keeps pinging the old nodes until the network
disconnect resolves, then rejoins.

-Zach

On Tuesday, February 19, 2013 9:48:28 AM UTC-5, Ophir Michaeli wrote:

Hi,

It is not clear from the documentation why minimum_master_nodes should be
higher than 1 for more than 2 nodes, and what are the considerations to
configure properly.

Thanks

On Tuesday, February 19, 2013 3:17:14 PM UTC+2, Zachary Tong wrote:

Elasticsearch will automatically elect a new master if the old one
becomes unavailable, but master's are not necessary for search/indexing
operations. E.g. any node in the cluster is fully capable of servicing a
search request, not just the master node.

The master is in charge of managing cluster metadata and state (mappings,
new indices, etc), in addition to normal indexing/search requirements.

It is also important to have your minimum_master_nodes setting
configured properly, to help avoid split-brains.

-Zach

On Tuesday, February 19, 2013 7:59:53 AM UTC-5, Ophir Michaeli wrote:

Hi,

What's the best practice for an elasticsearch client to handle a
scenario in which elasticsearch master falls?

From my tests I can connect to any of the slaves and continue on. Does
it matter that the client connect to a slave and not to a master?

Thanks,
Ophir

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Just to make sure I understood correctly, if I run 4 elasticsearch nodes (a
cluster) I'll set minimum_master_nodes to 3?

Because if one falls (including a case where the master falls) then the
remaining 3 can have a new master elected (they are 3 and
minimum_master_nodes is 3),

while the solitary node has only 1 node (itself) so it will wait till
reconnection to the others in the cluster.

On Tuesday, February 19, 2013 9:06:01 PM UTC+2, Zachary Tong wrote:

The recommended value is usually (N/2 + 1) where N is the number of
master-eligible nodes.

For example, if you have min_master_nodes == 1 in a three node cluster,
you can easily get a "split-brain". Imagine one node drops out of the
cluster, but remains active (e.g. the server didn't die, but the network
connection between it and the rest of the nodes died). This solitary node
sees the other two departing and thinks it is the only survivor, so it
elects itself as master and forms a new cluster.

Meanwhile, the other two nodes see the single node leave and just carry on
as usual.

Now, when the network disconnect resolves itself, the third node does not
attempt to rejoin the cluster because it thinks it is the cluster and
the other two are missing. This is a split-brain, where you have two
clusters when only one should exist. Even worse, they don't realize there
are two clusters so now you have rapidly diverging data states because they
aren't trying to remerge.

Min_master_nodes prevents this. If min_master_nodes == 2 in the above
scenario, the solitary node sees only one master eligible node (itself),
which is less than the required 2 nodes. Therefore, instead of electing
itself as master, it simply keeps pinging the old nodes until the network
disconnect resolves, then rejoins.

-Zach

On Tuesday, February 19, 2013 9:48:28 AM UTC-5, Ophir Michaeli wrote:

Hi,

It is not clear from the documentation why minimum_master_nodes should be
higher than 1 for more than 2 nodes, and what are the considerations to
configure properly.

Thanks

On Tuesday, February 19, 2013 3:17:14 PM UTC+2, Zachary Tong wrote:

Elasticsearch will automatically elect a new master if the old one
becomes unavailable, but master's are not necessary for search/indexing
operations. E.g. any node in the cluster is fully capable of servicing a
search request, not just the master node.

The master is in charge of managing cluster metadata and state
(mappings, new indices, etc), in addition to normal indexing/search
requirements.

It is also important to have your minimum_master_nodes setting
configured properly, to help avoid split-brains.

-Zach

On Tuesday, February 19, 2013 7:59:53 AM UTC-5, Ophir Michaeli wrote:

Hi,

What's the best practice for an elasticsearch client to handle a
scenario in which elasticsearch master falls?

From my tests I can connect to any of the slaves and continue on. Does
it matter that the client connect to a slave and not to a master?

Thanks,
Ophir

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Yep, that's correct. =)

-Zach

On Wednesday, February 20, 2013 4:33:54 AM UTC-5, Ophir Michaeli wrote:

Just to make sure I understood correctly, if I run 4 elasticsearch nodes
(a cluster) I'll set minimum_master_nodes to 3?

Because if one falls (including a case where the master falls) then the
remaining 3 can have a new master elected (they are 3 and
minimum_master_nodes is 3),

while the solitary node has only 1 node (itself) so it will wait till
reconnection to the others in the cluster.

On Tuesday, February 19, 2013 9:06:01 PM UTC+2, Zachary Tong wrote:

The recommended value is usually (N/2 + 1) where N is the number of
master-eligible nodes.

For example, if you have min_master_nodes == 1 in a three node cluster,
you can easily get a "split-brain". Imagine one node drops out of the
cluster, but remains active (e.g. the server didn't die, but the network
connection between it and the rest of the nodes died). This solitary node
sees the other two departing and thinks it is the only survivor, so it
elects itself as master and forms a new cluster.

Meanwhile, the other two nodes see the single node leave and just carry
on as usual.

Now, when the network disconnect resolves itself, the third node does not
attempt to rejoin the cluster because it thinks it is the cluster and
the other two are missing. This is a split-brain, where you have two
clusters when only one should exist. Even worse, they don't realize there
are two clusters so now you have rapidly diverging data states because they
aren't trying to remerge.

Min_master_nodes prevents this. If min_master_nodes == 2 in the above
scenario, the solitary node sees only one master eligible node (itself),
which is less than the required 2 nodes. Therefore, instead of electing
itself as master, it simply keeps pinging the old nodes until the network
disconnect resolves, then rejoins.

-Zach

On Tuesday, February 19, 2013 9:48:28 AM UTC-5, Ophir Michaeli wrote:

Hi,

It is not clear from the documentation why minimum_master_nodes should
be higher than 1 for more than 2 nodes, and what are the considerations to
configure properly.

Thanks

On Tuesday, February 19, 2013 3:17:14 PM UTC+2, Zachary Tong wrote:

Elasticsearch will automatically elect a new master if the old one
becomes unavailable, but master's are not necessary for search/indexing
operations. E.g. any node in the cluster is fully capable of servicing a
search request, not just the master node.

The master is in charge of managing cluster metadata and state
(mappings, new indices, etc), in addition to normal indexing/search
requirements.

It is also important to have your minimum_master_nodes setting
configured properly, to help avoid split-brains.

-Zach

On Tuesday, February 19, 2013 7:59:53 AM UTC-5, Ophir Michaeli wrote:

Hi,

What's the best practice for an elasticsearch client to handle a
scenario in which elasticsearch master falls?

From my tests I can connect to any of the slaves and continue on. Does
it matter that the client connect to a slave and not to a master?

Thanks,
Ophir

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.