HHello everyone,
I noticed something I cannot explain:
- I deploy Elasticsearch (following the quickstart guide) with the DeleteOnScaledownAndClusterDeletion policy.
- Then I patch the policy to DeleteOnScaledownOnly and immediately remove Elasticsearch:
kubectl -v 8 -n ${NAMESPACE} patch elasticsearch ${ES_NAME} -p '{"spec": {"volumeClaimDeletePolicy": "DeleteOnScaledownOnly"}}' --type=merge && kubectl -v 8 delete elasticsearch ${ES_NAME} -n ${NAMESPACE} - I expect that the Elasticsearch PVC will remain, but it is removed.
- If I wait 5 seconds between patching and removing, the PVC will NOT be removed:
kubectl -n ${NAMESPACE} patch elasticsearch ${ES_NAME} -p '{"spec": {"volumeClaimDeletePolicy": "DeleteOnScaledownOnly"}}' --type=merge && sleep 5 && kubectl delete elasticsearch ${ES_NAME} -n ${NAMESPACE}
After patching, I can see that the configuration has been applied:
kubectl -n ${NAMESPACE} get elasticsearch ${ES_NAME} -o json | jq > elastic_before.json
kubectl -n ${NAMESPACE} patch elasticsearch ${ES_NAME} -p '{"spec": {"volumeClaimDeletePolicy": "DeleteOnScaledownOnly"}}' --type=merge
kubectl -n ${NAMESPACE} get elasticsearch ${ES_NAME} -o json | jq > elastic_new.json
kubectl delete elasticsearch ${ES_NAME} -n ${NAMESPACE}
diff elastic_before.json elastic_new.json
8c8
< "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"elasticsearch.k8s.elastic.co/v1\",\"kind\":\"Elasticsearch\",\"metadata\":{\"annotations\":{},\"labels\":{\"branch\":\"feature-ftr-1737\",\"commit\":\"9080ffa4\"},\"name\":\"feature-ftr-1737\",\"namespace\":\"my-ns\"},\"spec\":{\"image\":\"registry.gitlab.com/nnikushkin_group/elastic/elasticsearch:8.14.3\",\"nodeSets\":[{\"config\":{\"node.store.allow_mmap\":false},\"count\":1,\"name\":\"master\",\"podTemplate\":{\"metadata\":{\"labels\":{\"branch\":\"feature-ftr-1737\",\"commit\":\"9080ffa4\",\"egress-frontoffice-policy\":\"allow\",\"ingress-controller-frontoffice-policy\":\"allow\",\"tier\":\"vertrouwd\"}},\"spec\":{\"containers\":[{\"env\":[{\"name\":\"ES_PASSWORD\",\"valueFrom\":{\"secretKeyRef\":{\"key\":\"elastic\",\"name\":\"feature-ftr-1737-es-elastic-user\"}}}],\"imagePullPolicy\":\"Never\",\"name\":\"elasticsearch\",\"resources\":{\"limits\":{\"cpu\":2,\"memory\":\"2Gi\"},\"requests\":{\"cpu\":2,\"memory\":\"2Gi\"}}}]}}}],\"version\":\"8.14.3\",\"volumeClaimDeletePolicy\":\"DeleteOnScaledownAndClusterDeletion\"}}\n"
---
> "kubectl.kubernetes.io/last-applied-configuration": "{\"apiVersion\":\"elasticsearch.k8s.elastic.co/v1\",\"kind\":\"Elasticsearch\",\"metadata\":{\"annotations\":{\"eck.k8s.elastic.co/orchestration-hints\":\"{\\\"no_transient_settings\\\":true,\\\"service_accounts\\\":true,\\\"desired_nodes\\\":{\\\"version\\\":2,\\\"hash\\\":\\\"2863175271\\\"}}\",\"elasticsearch.k8s.elastic.co/cluster-uuid\":\"RzLqgzU7T0SYgJDmd-Y5jw\"},\"creationTimestamp\":\"2024-08-15T10:29:17Z\",\"generation\":2,\"labels\":{\"branch\":\"feature-ftr-1737\",\"commit\":\"9080ffa4\"},\"name\":\"feature-ftr-1737\",\"namespace\":\"my-ns\",\"resourceVersion\":\"129001\",\"uid\":\"3c04c170-34dd-467a-8fc3-b825ba2c3562\"},\"spec\":{\"auth\":{},\"http\":{\"service\":{\"metadata\":{},\"spec\":{}},\"tls\":{\"certificate\":{}}},\"image\":\"registry.gitlab.com/nnikushkin_group/elastic/elasticsearch:8.14.3\",\"monitoring\":{\"logs\":{},\"metrics\":{}},\"nodeSets\":[{\"config\":{\"node.store.allow_mmap\":false},\"count\":1,\"name\":\"master\",\"podTemplate\":{\"metadata\":{\"creationTimestamp\":null,\"labels\":{\"branch\":\"feature-ftr-1737\",\"commit\":\"9080ffa4\",\"egress-frontoffice-policy\":\"allow\",\"ingress-controller-frontoffice-policy\":\"allow\",\"tier\":\"vertrouwd\"}},\"spec\":{\"containers\":[{\"env\":[{\"name\":\"ES_PASSWORD\",\"valueFrom\":{\"secretKeyRef\":{\"key\":\"elastic\",\"name\":\"feature-ftr-1737-es-elastic-user\"}}}],\"imagePullPolicy\":\"Never\",\"name\":\"elasticsearch\",\"resources\":{\"limits\":{\"cpu\":\"2\",\"memory\":\"2Gi\"},\"requests\":{\"cpu\":\"2\",\"memory\":\"2Gi\"}}}]}}}],\"transport\":{\"service\":{\"metadata\":{},\"spec\":{}},\"tls\":{\"certificate\":{},\"certificateAuthorities\":{}}},\"updateStrategy\":{\"changeBudget\":{}},\"version\":\"8.14.3\",\"volumeClaimDeletePolicy\":\"DeleteOnScaledownOnly\"},\"status\":{\"availableNodes\":1,\"conditions\":[{\"lastTransitionTime\":\"2024-08-15T10:29:59Z\",\"status\":\"True\",\"type\":\"ReconciliationComplete\"},{\"lastTransitionTime\":\"2024-08-15T10:29:19Z\",\"message\":\"All nodes are running version 8.14.3\",\"status\":\"True\",\"type\":\"RunningDesiredVersion\"},{\"lastTransitionTime\":\"2024-08-15T10:29:59Z\",\"message\":\"Service my-ns/feature-ftr-1737-es-internal-http has endpoints\",\"status\":\"True\",\"type\":\"ElasticsearchIsReachable\"},{\"lastTransitionTime\":\"2024-08-15T10:29:19Z\",\"message\":\"Successfully calculated compute and storage resources from Elasticsearch resource generation 2\",\"status\":\"True\",\"type\":\"ResourcesAwareManagement\"}],\"health\":\"green\",\"inProgressOperations\":{\"downscale\":{\"lastUpdatedTime\":\"2024-08-15T10:29:19Z\"},\"upgrade\":{\"lastUpdatedTime\":\"2024-08-15T10:29:19Z\"},\"upscale\":{\"lastUpdatedTime\":\"2024-08-15T10:29:19Z\"}},\"observedGeneration\":2,\"phase\":\"Ready\",\"version\":\"8.14.3\"}}\n"
11c11
< "generation": 2,
---
> "generation": 3,
18c18
< "resourceVersion": "129001",
---
> "resourceVersion": "129167",
101c101
< "volumeClaimDeletePolicy": "DeleteOnScaledownAndClusterDeletion"
---
> "volumeClaimDeletePolicy": "DeleteOnScaledownOnly"
124,125c124,125
< "lastTransitionTime": "2024-08-15T10:29:19Z",
< "message": "Successfully calculated compute and storage resources from Elasticsearch resource generation 2",
---
> "lastTransitionTime": "2024-08-15T10:31:53Z",
> "message": "Successfully calculated compute and storage resources from Elasticsearch resource generation 3",
142c142
< "observedGeneration": 2,
---
> "observedGeneration": 3,
However, if I repeat the same actions but use kubectl apply instead of kubectl patch, it works as expected (PVC is NOT removed)
kubectl -n ${NAMESPACE} get elasticsearch ${ES_NAME} -o yaml > elastic_before.yaml
cp -p elastic_before.yaml elastic_to_apply.yaml
sed -i "s|volumeClaimDeletePolicy: DeleteOnScaledownAndClusterDeletion|volumeClaimDeletePolicy: DeleteOnScaledownOnly|g" elastic_to_apply.yaml
kubectl -v 8 -n ${NAMESPACE} apply -f elastic_to_apply.yaml && kubectl -v 8 delete elasticsearch ${ES_NAME} -n ${NAMESPACE}
Is this behavior related to Kubernetes logic or ECK?
I also tried checking the patch and apply commands with -v=8, but it did not help me understand the behavior.
Thank you!