Our local ES cluster is running in k8s env, with 3 client/ingest pods, 3 data pods and 3 master pods. Previously we only assigned the persistent volume to data pods, now we think better to assign persistent volume to master pods also. We need complete this upgrade without service break, so want to know if there have any existing case or tips can be re-used.
Our current thinking is like below, but not sure if it can work or not, or still have some important steps need follow.
- Deploy a master statefulset with one node, let that new master node join existing master cluster to replicate the data from old master nodes.
- After the new master node joined the cluster and completed the data sync(how to know that?), remove an old master node
- Repeat steps 1,2 to replace all old master nodes with new master nodes with PV.
Any comments or suggestions are welcomed, Thanks!