Adding new master nodes to old cluster

Elastic Stack Elasticsearch
1

I have a production cluster with 4 master+Data node under X-pack Plat licence
We planning to split master and data node role, so i will add 3 master only node and set the master/data node to data node only
so my question is how to do this without without downtime

2

I suppose that you have replication enabled. In that case I'd do the following:

  • setup discovery.zen.minimum_master_nodes=3 https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery-zen.html (I think that you have wrong config right now since you have 4 master eligible nodes)
  • shutdown 1st data/master node and configure it to have only data role
  • wait for green cluster
  • shutdown 2nd data/master node and configure it to have only data role
  • start 1st master only node
  • wait for green cluster
  • check that you have 3 master eligible nodes
  • shutdown 3rd data/master node and configure it to have only data role
  • start 2nd master only node
  • wait for green cluster
  • check that you have 3 master eligible nodes
  • shutdown 4th data/master node and configure it to have only data role
  • start 3rd master only node
  • wait for green cluster
  • check that you have 3 master eligible nodes

Please check the settings before any modification so you do not end up without master node

1 Like
3

If you have a Platinum licence for this cluster then you are eligible for support. Please get hold of your support contact who should be happy to guide you through this process.

I would do roughly what @mats990 said, except that I'd start one of the new dedicated master nodes first. With discovery.zen.minimum_master_nodes: 3 you can have up to 5 master-eligible nodes, and the more of them you have the more resilient you are to failures, so I'd try and do the process with 4 or 5 master nodes online instead of 3 or 4.

1 Like
4

ok thank you yes but i prefer help elastic community :slight_smile: and i like this discuss !
so if i understand :
i'd start with the new master with
discovery.zen.ping.unicast.hosts: ["oldmasterdata1", "oldmasterdata2", "oldmasterdata3", "newmasteronly", "newmasteronly2", "newmasteronly3"]
then start oldmasterdata1 change conf with the same discovery.zen.ping.unicast.hosts "like the newmaster"

oldmasterdata1 : configure and restart it
newmasteronly2: configure and restart it
oldmasterdata2: configure and restart it
newmasteronly3: configure and restart it
oldmasterdata3: configure and restart it
oldmasterdata4: configure and restart it

..........and at the end re-change discovery.zen.minimum_master_nodes : 2 and
discovery.zen.ping.unicast.hosts: [ "newmasteronly", "newmasteronly2", "newmasteronly3"]
then restart all nodes ?
that's right ?

5

no response ?

6

It's hard to say, there's lots of detail missing in your post. How will you be reconfiguring each node? How will you be changing discovery.zen.minimum_master_nodes throughout? It sounds like you're going along the right path, but there's a risk of getting things wrong in this process and losing data as a result.

7

sorry yes it'snot clear .
i have 3 new master (with no config)

and 4 nodes (3 data/master plus 1 data node)
i want to add the new 3 master and the change the old node to data only
and to have as result discovery.zen.minimum_master_nodes : 2

Thnx !

8

ok i will create a case ....

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.