Migrate node from master + data to data only

Elastic Search
1

we are running 5 node Elasticsearch cluster with 3 master and 3 data node
one node is acting as both master + data ()
This is my cluster configuration

xx.yy.zz.qq 53 93 7 1.58 1.00 1.10 hilrstw - elsdata02
xx.yy.zz.qq 16 95 37 6.86 7.21 6.95 cirs - elsdata03
xx.yy.zz.qq 6 10 95 0 0.03 0.01 0.00 ilmr - elsmstr02
xx.yy.zz.qq 31 86 28 5.33 5.61 6.01 cdhilmrstw * elsdata01
xx.yy.zz.qq 15 95 6 0.16 0.04 0.03 ilmr - elsmstr01

Elastic search version is 7.10.0

  1. whats the best way to migrate migrate the elsdata01 node which is main master node from master + data to only data.

  2. would this need a full cluster restart or could i manage with rolling restarts

  3. Currently in my es.yml I have cluster bootstrapping set to the 3 current master nodes, from reading up on this setting it's only used for the very first time, i take it i dont need this anymore even if onbaording brand new masters?

2

Which other node will receive the master role?

If you remove the master role from this node, you will have only two other nodes with a master role, you need at least 3 to have resiliency.

Are you going to add other node to act as master or will add the master role to other node?

3

@leandrojmp Thanks for the reply , i have plan to add 1 more node as master , i am confused if i can add that master node to cluster before converting master + data to data node or after converting master + data node to data node and what is best approach to convert the master + data node to data node without impacting the cluster as its production cluster

4

You should add another master eligible node before changing the roles of a node.

To convert a data + master to a data only node you will need to change the configuration in elasticsearch.yml to remove the master role and restart the node.

But you should add another master eligible master before doing that, as this would leave ir cluster on a state where if one of the other masters eligible has some issue, the entire cluster will be down.

1 Like
5

Thanks @leandrojmp thanks for the update i have one more question related to cluster.initial_master_nodes settings on other nodes which are part of cluster, as i am migrating data + master to data , do i have to remove this node from all other nodes before starting the migration or add new master to all nodes cluster.initial_master_nodes before starting the conversion of the data + master node to data node

6

Once a cluster is formed you need to remove cluster.initial_master_nodes from the elasticsearch.yml of all nodes, and you also should not add this setting in new nodes joining the cluster.

This is mentioned here.

After the cluster has formed, remove the cluster.initial_master_nodes setting from each node’s configuration and never set it again for this cluster. Do not configure this setting on nodes joining an existing cluster. Do not configure this setting on nodes which are restarting. Do not configure this setting when performing a full-cluster restart.

You just need to have the discovery.seed_hosts setting with all the master-eligible nodes in the cluster.

7

@leandrojmp thanks a ton

Oh in that case we first need to remove this cluster.initial_master_nodes settings from all nodes , can i do that in rolling restart manner , like change at one node at a time and then restart , is any preferred order to update the settings like start with master , then move to data node ?

also we have discovery.seed_hosts settings consist of all nodes include master and data node , shall i change it also before hand then like before change cluster.initial_master_nodes settings ? also can i do change in rolling restart manner , like change at one node at a time and then restart , is any preferred order to update the settings like start with master , then move to data node ?

8

The setting discovery.seed_hosts needs to have only nodes that are master-eligible, no data nodes.

You should be able to do this with a rolling restart, I would first add the new master node with the updated discovery.seed_hosts containing the 3 nodes that will be master eligible, after that you can change the settings and restart the other nodes, including removing the master role from your current data + master node, but to be safe I would also set up a maintenance window in the case something goes wrong and you need to do a full cluster restart.

When you stop the current master, that you need to change the roles, your cluster will elect a new master.

10

Hi @leandrojmp Thanks for update
Please confirm if below procedure looks ok as overall operation , also what initial operation we can do to avoid data loss or risk to cluster stability

  1. Add new master node to the cluster with the discovery.seed_hosts contains all 4 nodes (3 current master + new master node which is new master node IP)
  2. Modify the discovery.seed_host across all nodes in the cluster with details as (3 current master + new master node which is new master node IP)
    This operation we can do with rolling restart with each node at a time
  3. Disable the cluster.initial_master_nodes across all nodes in the cluster
    This operation we can do with rolling restart with each node at a time

    4.Modify the main master + data node to data node .
    This operation we can do with modification of role and then just restart of the node .
  4. Modify the discovery.seed_host across all nodes in the cluster with details as (3 new master - old master node which is data node now)
    This operation we can do with rolling restart with each node at a time
11

Hi @leandrojmp
sorry one more question
do you see any issue if i add one extra master node to existing cluster that make total master to be 4 , like breaking some quoram ?
Any additional step or operation i need to do to make cluster stable ?

12

TBH if you only have 3 data nodes you almost certainly don't need 3 separate master nodes. It's just making your life difficult. I'd strongly recommend you change your cluster to 3 do-everything nodes.

It won't make any difference if you have 3 or 4 master nodes. No need to do anything special.

13

This needs to be step zero btw, not step 3. Do it right now, if not sooner. See e.g. these docs:

After the cluster has formed, remove the cluster.initial_master_nodes setting from each node’s configuration and never set it again for this cluster. Do not configure this setting on nodes joining an existing cluster. Do not configure this setting on nodes which are restarting. Do not configure this setting when performing a full-cluster restart.

14

Thanks @DavidTurner i am planning operation in next to next weekend , i will remove the cluster.initial_master_nodes first on data node first and then master node , hopefully order is ok in removal of the settings cluster.initial_master_nodes.

15

In this case will cluster works with DO nodes , no master ?
i also have kibana running on master nodes that why want to run master seprately. My concerns is if i add 4th master as temporary state will it impact cluster ? as next step for me to remove the data + master node to data only node