# How to migrate master nodes to use persistent volume in kubernetes env **URL:** \ **Category:** Elasticsearch\ **Created:** [December 3, 2018, 10:20am UTC](https://discuss.elastic.co/t/how-to-migrate-master-nodes-to-use-persistent-volume-in-kubernetes-env/159140 "2018-12-03T10:20:47Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![wangywy](https://avatars.discourse-cdn.com/v4/letter/w/0ea827/32.png) [@wangywy](https://discuss.elastic.co/u/wangywy)\ **Post date:** [December 3, 2018, 10:20am UTC](https://discuss.elastic.co/t/how-to-migrate-master-nodes-to-use-persistent-volume-in-kubernetes-env/159140/1 "2018-12-03T10:20:47Z")
Hi all, Our local ES cluster is running in k8s env, with 3 client/ingest pods, 3 data pods and 3 master pods. Previously we only assigned the persistent volume to data pods, now we think better to assign persistent volume to master pods also. We need complete this upgrade without service break, so want to know if there have any existing case or tips can be re-used. Our current thinking is like below, but not sure if it can work or not, or still have some important steps need follow. 1. Deploy a master statefulset with one node, let that new master node join existing master cluster to replicate the data from old master nodes. 2. After the new master node joined the cluster and completed the data sync(how to know that?), remove an old master node 3. Repeat steps 1,2 to replace all old master nodes with new master nodes with PV. Any comments or suggestions are welcomed, Thanks! ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [December 7, 2018, 10:59pm UTC](https://discuss.elastic.co/t/how-to-migrate-master-nodes-to-use-persistent-volume-in-kubernetes-env/159140/2 "2018-12-07T22:59:23Z")
What you describe seems reasonable, but you must make sure that `discovery.zen.minimum_master_nodes` is [strictly greater than half of the number of master-eligible nodes in your cluster](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#split-brain) at all times to avoid the risk of split-brain. This means that you must increase `discovery.zen.minimum_master_nodes` to 3 before you bring the fourth node into the cluster. > [@wangywy](#): > > After the new master node joined the cluster and completed the data sync(how to know that?), remove an old master node I think if it's a dedicated master node then as long as it has fully joined the cluster it has everything it needs for you to carry on. ---
**Author:** ![wangywy](https://avatars.discourse-cdn.com/v4/letter/w/0ea827/32.png) [@wangywy](https://discuss.elastic.co/u/wangywy)\ **Post date:** [December 10, 2018, 9:23am UTC](https://discuss.elastic.co/t/how-to-migrate-master-nodes-to-use-persistent-volume-in-kubernetes-env/159140/3 "2018-12-10T09:23:39Z")
Thanks for the help! If there have any sequence requirement? Like increase discovery.zen.minimum\_master\_nodes at first then add the new nodes? or add new node at first then increase discovery.zen.minimum\_master\_nodes? Also, when remove old master nodes, need decrease discovery.zen.minimum\_master\_nodes at first or can do that after removal? Thanks! ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [December 10, 2018, 10:15am UTC](https://discuss.elastic.co/t/how-to-migrate-master-nodes-to-use-persistent-volume-in-kubernetes-env/159140/4 "2018-12-10T10:15:01Z")
> [@wangywy](#): > > If there have any sequence requirement? Like increase discovery.zen.minimum\_master\_nodes at first then add the new nodes? or add new node at first then increase discovery.zen.minimum\_master\_nodes? You should increase `discovery.zen.minimum_master_nodes` first, because if you don't you will briefly have 4 master-eligible nodes and `discovery.zen.minimum_master_nodes: 2` which is incorrect. For the same reason, you should not decrease it until after removing the old node. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 7, 2019, 10:15am UTC](https://discuss.elastic.co/t/how-to-migrate-master-nodes-to-use-persistent-volume-in-kubernetes-env/159140/5 "2019-01-07T10:15:05Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.