How to migrate master nodes to use persistent volume in kubernetes env

[wangywy]

Hi all,

Our local ES cluster is running in k8s env, with 3 client/ingest pods, 3 data pods and 3 master pods. Previously we only assigned the persistent volume to data pods, now we think better to assign persistent volume to master pods also. We need complete this upgrade without service break, so want to know if there have any existing case or tips can be re-used.

Our current thinking is like below, but not sure if it can work or not, or still have some important steps need follow.

1. Deploy a master statefulset with one node, let that new master node join existing master cluster to replicate the data from old master nodes.
2. After the new master node joined the cluster and completed the data sync(how to know that?), remove an old master node
3. Repeat steps 1,2 to replace all old master nodes with new master nodes with PV.

Any comments or suggestions are welcomed, Thanks!

[DavidTurner]

What you describe seems reasonable, but you must make sure that discovery.zen.minimum_master_nodes is strictly greater than half of the number of master-eligible nodes in your cluster at all times to avoid the risk of split-brain. This means that you must increase discovery.zen.minimum_master_nodes to 3 before you bring the fourth node into the cluster.

@wangywy

After the new master node joined the cluster and completed the data sync(how to know that?), remove an old master node

I think if it's a dedicated master node then as long as it has fully joined the cluster it has everything it needs for you to carry on.

[wangywy]

Thanks for the help!

If there have any sequence requirement? Like increase discovery.zen.minimum_master_nodes at first then add the new nodes? or add new node at first then increase discovery.zen.minimum_master_nodes?

Also, when remove old master nodes, need decrease discovery.zen.minimum_master_nodes at first or can do that after removal?

Thanks!

[DavidTurner]

@wangywy

If there have any sequence requirement? Like increase discovery.zen.minimum_master_nodes at first then add the new nodes? or add new node at first then increase discovery.zen.minimum_master_nodes?

You should increase discovery.zen.minimum_master_nodes first, because if you don't you will briefly have 4 master-eligible nodes and discovery.zen.minimum_master_nodes: 2 which is incorrect. For the same reason, you should not decrease it until after removing the old node.