# Volume Expansion StatefulSet Recreation Logic **URL:** \ **Category:** Elastic Cloud on Kubernetes (ECK)\ **Created:** [May 14, 2021, 3:44pm UTC](https://discuss.elastic.co/t/volume-expansion-statefulset-recreation-logic/273041 "2021-05-14T15:44:25Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![zpear](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zpear/32/88772_2.png) [@zpear](https://discuss.elastic.co/u/zpear)\ **Post date:** [May 14, 2021, 3:44pm UTC](https://discuss.elastic.co/t/volume-expansion-statefulset-recreation-logic/273041/1 "2021-05-14T15:44:25Z")
Hi, I've been looking into how the ECK operator upsizes the persistent volumes being used by the ES cluster. Looking at the [PR that added this functionality](https://github.com/elastic/cloud-on-k8s/pull/3752/files#) and reading the comments I wanted to confirm my understanding is correct that the StatefulSets' recreation is coordinated with [special annotations](https://github.com/elastic/cloud-on-k8s/blob/master/pkg/controller/elasticsearch/driver/pvc_expansion.go#L28) added to the _same_ ES custom resource object that the operator listens to for performing cluster updates. If so, what stops a client from overwriting these annotations and interfering with the StatefulSet recreation? I was also wondering why this approach was taken rather than blocking to sequentially delete and recreate each relevant StatefulSet, avoiding using k8s annotations to coordinate recreation entirely? I haven't dug too deeply into the codebase so was hoping to get some pointers on where my understanding may be off. Thanks! ---
**Author:** ![pebrc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pebrc/32/101790_2.png) [@pebrc](https://discuss.elastic.co/u/pebrc)\ **Post date:** [May 14, 2021, 4:11pm UTC](https://discuss.elastic.co/t/volume-expansion-statefulset-recreation-logic/273041/2 "2021-05-14T16:11:56Z")
> [@zpear](#): > > If so, what stops a client from overwriting these annotations and interfering with the StatefulSet recreation? Nothing. However the stateful set would still be recreated eventually based on the Elasticsearch spec. I would consider this a malicious client and we cannot guarantee correct operation in this case. The motivation for the annotation based approach stems from this comment: > > > Add support for resizing Elasticsearch volumes, by simply editing the storage re…quests in the \`volumeClaimTemplates\` section of the spec. > > \*\*Demo\*\* > > !\[assembled\](https://user-images.githubusercontent.com/1050393/93190702-f431df00-f743-11ea-9f81-2abddf6c0d4b.gif) > > \*\*Constraints\*\* > > \* Volume size can only be increased, not decreased (k8s limitation, not likely to change anytime soon) > \* Only possible if the storage class specifies \`allowVolumeExpansion: true\` > \* Hot filesystem resize without re-creating the Pod is \[possible in many cases\](https://kubernetes.io/blog/2018/07/12/resizing-persistent-volumes-using-kubernetes/#online-file-system-expansion) (GCE-PD, AWS-EBS, Azure, TopoLVM, etc.). If the storage driver does not allow it, it is necessary to \*manually\* delete the Pods. ECK does not do it. > > \*\*Implementation overview\*\* > > Resizing PVCs is \[not supported by the StatefulSet controller\](https://github.com/kubernetes/kubernetes/issues/68737) at this time. The volume section of the StatefulSet spec is immutable. > However, it is possible to work around the StatefulSet limitations this way: > 1. Update the storage requests spec of all existing PVCs: they are immediately resized by the storage driver, if inline expansion is supported. Otherwise Pods need to be recreated. > 2. We now have resized volumes, but the StatefulSet still specifies the old volume size. Any new replica will end up with a smaller PVC, that in turn needs to be resized. > 3. Delete the StatefulSet (but not the Pods it owns), then recreate it with the new volume spec. > 4. Address the case where Pods may become orphans if something happens while the StatefulSet is getting recreated, by ensuring we always recreate deleted StatefulSets, and ensure we set a temporary ownerRef on the Pods while they are not owned by the deleted StatefulSet anymore. > > \*\*Follow-up (in other PRs)\*\* > > \- \[x\] Add E2E tests to cover volume expansion: https://github.com/elastic/cloud-on-k8s/pull/3769 > \- \[x\] Write documentation: https://github.com/elastic/cloud-on-k8s/pull/3797 > \- (later) Figure out a way to automatically recreate Pods if inline filesystem expansion is not supported: https://github.com/elastic/cloud-on-k8s/issues/3757 > \- Check whether storage class is compatible with volume expansion: https://github.com/elastic/cloud-on-k8s/issues/3767 > > Relates https://github.com/elastic/cloud-on-k8s/issues/325. We want to avoid a situation where we end up with orphaned Pods that remain after a user decides to rename a node set (and thus make the now deleted stateful set obsolete) in the middle of the resizing process. Admittedly somewhat of an edge case but still possible. An earlier version of the implementation deleted and recreated the StatefulSets directly. Another reason we don't do that in a blocking way is that we cannot guarantee that the operator will not be interrupted in the middle (e.g. manual restart, OOM, operator upgrade etc). Any algorithm we use needs to be resilient against that. Also we try to avoid long blocking operations in the operator as it reduces the operators ability to process changes for other Elasticsearch clusters while we are blocking. Hope that makes sense! ---
**Author:** ![zpear](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/zpear/32/88772_2.png) [@zpear](https://discuss.elastic.co/u/zpear)\ **Post date:** [May 14, 2021, 4:18pm UTC](https://discuss.elastic.co/t/volume-expansion-statefulset-recreation-logic/273041/3 "2021-05-14T16:18:27Z")
Fantastic, thank you for the quick and thorough reply -- that makes sense to me! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 11, 2021, 4:18pm UTC](https://discuss.elastic.co/t/volume-expansion-statefulset-recreation-logic/273041/4 "2021-06-11T16:18:36Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.