Introduction of dedicated master nodes

Elastic Stack Elasticsearch
1

Hello,

As recommended, we want to make our cluster more stable by adding dedicated master nodes.
Actualy, we have 5 nodes that do all data with only first 3 to be master eligible.

So, we have 3 new nodes that we want to dedicate to be master eligibles and do only that work.

Is there something important in making this work:

  • shutdown cluster
  • modify actual 5 nodes with master: false flag and data: true
  • make 3 new nodes master:true and data: false
  • modify all nodes to discover using 3 new master nodes addresses
  • we can optionnally disable http port on master nodes to make them not receiving REST requests.
  • start cluster

Is it that simple?
Actual master don't store anything that the new one will need to be aware?

Thanks
Bruno Lavoie

2

There's no need to shut down the cluster completely. Just spin up the new master-only nodes and make sure they join the cluster, then do a rolling restart of the old ones to change their configurations.

2 Likes
3

Ok ,

With the new nodes with master : true, the number of master eligible is then bumped to a total of 6 while the original discovery.zen.minimum_master_nodes setting is 2.

Is there a chance to have a split brain or something like this? Should we change discovery.zen.minimum_master_nodes setting temporarily? When rolling restart, do we should restart current master first or last?

Thanks again

4

I replied but not to you, I reply again just to be sure you'll be notified.
Thanks

5

Is there a chance to have a split brain or something like this?

Ah, good point. Theoretically yes, for exactly the reason you mention.

When rolling restart, do we should restart current master first or last?

I'd say last so that you're guaranteed a single master reelection.

6

Hello Magnus,

Ok, so good... I think I have a minimum understanding of ES. :slightly_smiling:

And to be safe, I'll also adjust minimum_master_nodes transient setting while adding and removing nodes.

Again, thanks for your help.

7

Hi Bruno,
I need to do the same thing as you. I have 2 data/master nodes and trying to add 3 master nodes. Master_min_nodes is 3. When I start new masters, they don't join existing cluster. Only when I restart 2 master/data nodes, they join the 3 master node cluster. Did you have such problem? Your new master nodes joining your original cluster?
Thanks.

8

Not got such a problem... sorry...
About you new nodes, not joining as master nodes, rather are they joining as another type node (client, data, ...)?

9

They don't join the existing 2 node cluster. But rather, when I start 3 new master nodes, they form their own 3 master node cluster.

10

be sure that they are part of the same cluster (name) and that no firewalls blocking...

11

Do the new master nodes have the IP addresses of the old nodes in their unicast configuration? Are all nodes on the same version of Elasticsearch?

1 Like
12

They all have the same discovery.zen.ping.unicast.hosts. It is the list of all 5 nodes (2 original plus 3 new master nodes). They are all on the same version. My issue is that new master nodes are not joining existing cluster. If I shutdown 2 node original cluster and restart, it does join the 3 master node cluster. Why it doesn't work the other way, beats me. I have different issues I run into when my 2 original nodes join 3 master node (created a topic here but no answers yet on that one).

On my 3 new masters, I set:
gateway.expected_master_nodes: 3
gateway.minimum_master_nodes: 2
gateway.expected_nodes: 5

On my original 2 node cluster, I have:
gateway.expected_nodes: 2
gateway.expected_master_nodes: 3

13

Very, very good point!