# Pods not finding PVC when using storage class with Retain reclaim policy **URL:** \ **Category:** Elastic Cloud on Kubernetes (ECK)\ **Created:** [June 14, 2021, 1:08pm UTC](https://discuss.elastic.co/t/pods-not-finding-pvc-when-using-storage-class-with-retain-reclaim-policy/275835 "2021-06-14T13:08:28Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![pkaramol](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pkaramol/32/22610_2.png) [@pkaramol](https://discuss.elastic.co/u/pkaramol)\ **Post date:** [June 14, 2021, 1:08pm UTC](https://discuss.elastic.co/t/pods-not-finding-pvc-when-using-storage-class-with-retain-reclaim-policy/275835/1 "2021-06-14T13:08:28Z")
I am using in an `Elasticsearch` instance, a `StorageClass` that has `Retain` (instead of `Delete`) as its reclaim policy. Here are my `PVC`s **before** deleting the `Elasticsearch` instance ```auto ▶ k get pvc NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE elasticsearch-data--es-multirolenodes1-0 Bound pvc-ba157213-67cf-4b81-8fe2-6211b771e62c 20Gi RWO balanced-retain-csi 8m15s elasticsearch-data--es-multirolenodes1-1 Bound pvc-e77dbb00-7cad-419f-953e-f3398e3860f4 20Gi RWO balanced-retain-csi 7m11s elasticsearch-data--es-multirolenodes1-2 Bound pvc-b258821b-0d93-4ea3-8bf1-db590b93adfd 20Gi RWO balanced-retain-csi 6m5s ``` I deleted and re-installed the `helm` chart with the hope that due to the `Retain` policy, the new pods (i.e. their `PVC`s would bind to the existing `PV`s (and data wouldn't get lost) However now my pods of the `nodeSet` are all in pending state with this error ```auto Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 2m37s default-scheduler persistentvolumeclaim "elasticsearch-data--es-multirolenodes1-0" is being deleted Normal NotTriggerScaleUp 2m32s cluster-autoscaler pod didn't trigger scale-up: 2 persistentvolumeclaim "elasticsearch-data--es-multirolenodes1-0" not found Warning FailedScheduling 12s (x7 over 2m37s) default-scheduler persistentvolumeclaim "elasticsearch-data--es-multirolenodes1-0" not found ``` Why is this happening? Is there a way to save the data when the `Elasticsearch` resource is accidentally deleted (and recreated with the **exact** same configuration) ? **edit** : Here are the corresponding `PV`s ```auto ▶ k get pv pvc-b258821b-0d93-4ea3-8bf1-db590b93adfd 20Gi RWO Retain Released elastic/elasticsearch-data--es-multirolenodes1-2 balanced-retain-csi 20m pvc-ba157213-67cf-4b81-8fe2-6211b771e62c 20Gi RWO Retain Released elastic/elasticsearch-data--es-multirolenodes1-0 balanced-retain-csi 22m pvc-e77dbb00-7cad-419f-953e-f3398e3860f4 20Gi RWO Retain Released elastic/elasticsearch-data--es-multirolenodes1-1 balanced-retain-csi 21m ``` There is of course no `PVC` now ```auto ▶ k get pvc No resources found in elastic namespace. ``` The `StorageClass` under consideration is using the `csi` driver for the GCP persistent disk, fwiw ---
**Author:** ![michael.morello](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michael.morello/32/47448_2.png) [@michael.morello](https://discuss.elastic.co/u/michael.morello)\ **Post date:** [June 15, 2021, 6:15am UTC](https://discuss.elastic.co/t/pods-not-finding-pvc-when-using-storage-class-with-retain-reclaim-policy/275835/2 "2021-06-15T06:15:23Z")
> [@pkaramol](#): > > Why is this happening? A PV can't be used until it is in an `Available` state, according to the [K8S documentation](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#retain): > When the PersistentVolumeClaim is deleted, the PersistentVolume still exists and the volume is considered "released". But it is not yet available for another claim because the previous claimant's data remains on the volume. An administrator can manually reclaim the volume with the following steps. You must delete/cleanup the `claimRef` to make them available again. > [@pkaramol](#): > > Is there a way to save the data when the `Elasticsearch` resource is accidentally deleted (and recreated with the **exact** same configuration) ? The primary way to backup Elasticsearch clusters are [snapshots](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-snapshots.html). Regarding Kubernetes/Custom resources we do not have recommendations at the moment, it might be tied to the way you backup other resources or your K8S control plane (with a CD pipeline using `git`, using `etcd` snapshots...), it's hard to give specific recommendations. Since ECK 1.5.0 you can also [decouple the lifecycle of the cluster from the lifecycle of the PVCs](https://www.elastic.co/guide/en/cloud-on-k8s/1.5/k8s-volume-claim-templates.html#k8s_controlling_volume_claim_deletion). We also provide a tool to [reattach pv](https://github.com/elastic/cloud-on-k8s/tree/master/support/reattach-pv) but it must be used only in last resort. ---
**Author:** ![pkaramol](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pkaramol/32/22610_2.png) [@pkaramol](https://discuss.elastic.co/u/pkaramol)\ **Post date:** [June 15, 2021, 10:03am UTC](https://discuss.elastic.co/t/pods-not-finding-pvc-when-using-storage-class-with-retain-reclaim-policy/275835/3 "2021-06-15T10:03:12Z")
Thanks for this elaborate answer. It is just weird that the same process, i.e. performing a `helm install` and then `helm delete` when using the [official](https://github.com/elastic/helm-charts) `helm` charts, seems to keep the PVC and I am trying to find out why this difference in the behaviour. ---
**Author:** ![pkaramol](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pkaramol/32/22610_2.png) [@pkaramol](https://discuss.elastic.co/u/pkaramol)\ **Post date:** [June 15, 2021, 10:49am UTC](https://discuss.elastic.co/t/pods-not-finding-pvc-when-using-storage-class-with-retain-reclaim-policy/275835/5 "2021-06-15T10:49:09Z")
No, it is ECK that I am using. I just noticed that - performing `helm delete` (when using the `helm` **charts** ) retains the PVCs - deleting the `Elasticsearch` resource (when using **ECK** ) removes the PVCs In any case, setting `volumeClaimDeletePolicy: DeleteOnScaledownOnly` seems to do the job since I deleted and re-created (with the exact same configuration) the `Elasticsearch` resource and there was an actual data retention (the PVC were not deleted and the indices were there when the new pods came up). ---
**Author:** ![pkaramol](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pkaramol/32/22610_2.png) [@pkaramol](https://discuss.elastic.co/u/pkaramol)\ **Post date:** [June 15, 2021, 10:51am UTC](https://discuss.elastic.co/t/pods-not-finding-pvc-when-using-storage-class-with-retain-reclaim-policy/275835/6 "2021-06-15T10:51:02Z")
ΒΤW it is not very obvious / intuitive what `volumeClaimDeletePolicy: DeleteOnScaledownOnly` means. In which case / scenario the PVCs will be **actually** deleted? (they were not deleted when deleting the `Elasticsearch` resource anyway. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 13, 2021, 10:51am UTC](https://discuss.elastic.co/t/pods-not-finding-pvc-when-using-storage-class-with-retain-reclaim-policy/275835/7 "2021-07-13T10:51:22Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.