Adding/removing master eligible nodes to/from existing cluster

Elastic Stack Elasticsearch
1

Hello,

Considering an existing ES cluster with 3 master eligible nodes, what is the recommended steps to run in order to add 3 new master eligible nodes:

  1. add the nodes to the cluster and then
  2. update discovery.zen.minimum_master_nodes
    or 2, followed by 1?

What about removing 3 master eligible nodes out of 6 ? Is the active master automatically re-elected; is it recommended to move the active master before removing the nodes from the cluster or ES handles it ? Is this a safe order:

  1. remove the nodes from the cluster and then
  2. update discovery.zen.minimum_master_nodes
    or the other way around ?

And what happens with the requests (index/query) during re-election ?

2

Do you not have minimum masters set now?

3

Yes, I have, 2. And I assume I need to keep (master_eligible_nodes / 2) + 1, right ?

But going to 6 master nodes, a quorum of 2 could potentially generate a split-brain, so I assume that minimum_master_nodes must be updated to 4.

And again, going from 6 to 3 nodes, if the minimum_master_nodes remains 4, a master election is not possible anymore even if all the master eligible nodes are up.

So basically, my 1st question is what's the right order of operations, if there is one, to scale-in/scale-out, and
2nd, what happens with the requests (index/query) during re-election

4

The best way would be to add all 3 new masters, then shut down one of the other nodes, set node.master: false and restart it, and then do that to the other two master eligible, allowing one of the new masters to take on the role.

5

I have 2 scenarios, basically Scale-in and Scale-out:

  1. going from 3 master eligible nodes to 6 master eligible nodes:
    You said "add all 3 new masters". Ok, but this implies 2 steps: adding the master eligible nodes into the cluster and updating the minimum_master_nodes in order to avoid split brain situations. Not clear for me in what order should I run these steps.

  2. going from 6 master eligible nodes down to 3 master eligible nodes.
    The concern explained for 1 is valid for this scenario as well.
    On top of that, if the current master active is one of the 3 nodes that I plan to remove ,then I'll have to move it , right ? This raised new questions:

  • what happens with indexing operations during the master election ? Assuming that I make sure there are no indices created during the re-election, is the cluster able to process index requests?
  • how do I handle master re-election ? Is shutting down the node the only option ?
6

Add the nodes first.
And unless you expect this process to take many hours or days, you can reasonably skip changing minimum masters, just don't waste time after adding the nodes and then updating the configs.

Yes to both.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.