Hi, i am trying to deploy a test elasticsearch cluster on a 5 worker nodes kubernetes cluster.
I am using 2 storage classes, one for data and for snapshot repositories as below.
---
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
namespace: elastic
name: quickstart
spec:
version: 8.11.3
secureSettings:
- secretName: snapshot-settings
nodeSets:
- name: default
count: 5
volumeClaimTemplates:
- metadata:
name: elasticsearch-data
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 25Gi
storageClassName: storage-default
- metadata:
name: elasticsearch-repo-data
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 50Gi
storageClassName: nfs-csi
podTemplate:
spec:
volumes:
- name: custom-truststore
secret:
secretName: custom-truststore
- name: elasticsearch-data
persistentVolumeClaim:
claimName: elasticsearch-data
- name: elasticsearch-repo-data
persistentVolumeClaim:
claimName: elasticsearch-repo-data
imagePullSecrets:
- name: itsmregcred
initContainers:
- name: sysctl
securityContext:
privileged: true
runAsUser: 0
command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
containers:
- name: elasticsearch
image: nexus.itsmtaxud.priv:9003/repository/eck/elasticsearch/elasticsearch:8.11.3
imagePullPolicy: IfNotPresent
volumeMounts:
- name: custom-truststore
mountPath: /usr/share/elasticsearch/config/custom-truststore
- name: elasticsearch-data
mountPath: /usr/share/elasticsearch/data
- name: elasticsearch-repo-data
mountPath: /repodata
env:
- name: path.repo
value: "/repodata"
- name: ES_PATH_REPO
value: "/repodata" # Set the path.repo to /repodata
- name: ES_JAVA_OPTS
value: "-Djavax.net.ssl.trustStore=/usr/share/elasticsearch/config/custom-truststore/cacerts -Djavax.net.ssl.keyStorePassword=xxxxxxxxxxxx"
The repo-data is configured with nfs storage class and the issue is when apply the elasticsearch file the operator is not creating only 1 PVC NFS type and mount it in all the elastic pods, but one for each node.
I tried to create initialy the PV and PVC and when i am applying the the elasticsearch file the operator is deleting the existing PVC and still recreates 5.
This is the PVC definition if i try to create it before:
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
namespace: elastic
name: elasticsearch-repo-data
labels:
common.k8s.elastic.co/type: elasticsearch
elasticsearch.k8s.elastic.co/cluster-name: quickstart
elasticsearch.k8s.elastic.co/statefulset-name: quickstart-es-default
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 50Gi
storageClassName: nfs-csi
And this is what elasticoperator log show:
{"log.level":"info","@timestamp":"2023-12-19T08:36:43.801Z","log.logger":"elasticsearch-controller","message":"Deleting PVC","service.version":"2.10.0+59c1e727","service.type":"eck","ecs.version":"1.4.0","iteration":"849","namespace":"elastic","es_name":"quickstart","namespace":"elastic","pvc_name":"elasticsearch-repo-data"}
{"log.level":"error","@timestamp":"2023-12-19T08:36:44.218Z","log.logger":"manager.eck-operator","message":"Reconciler error","service.version":"2.10.0+59c1e727","service.type":"eck","ecs.version":"1.4.0","controller":"elasticsearch-controller","object":{"name":"quickstart","namespace":"elastic"},"namespace":"elastic","name":"quickstart","reconcileID":"11408387-9a9a-45a7-85e4-56b029080e29","error":"while updating pvc during owner ref reconciliation: Operation cannot be fulfilled on persistentvolumeclaims \"elasticsearch-repo-data-quickstart-es-default-2\": the object has been modified; please apply your changes to the latest version and try again","errorCauses":[{"error":"while updating pvc during owner ref reconciliation: Operation cannot be fulfilled on persistentvolumeclaims \"elasticsearch-repo-data-quickstart-es-default-2\": the object has been modified; please apply your changes to the latest version and try again"}],"error.stack_trace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.2/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.2/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.2/pkg/internal/controller/controller.go:227"}
This is how the pvc output looks when i am deploying the elasticsearch file
kubectl get pvc -n elastic
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
elasticsearch-data-quickstart-es-default-0 Bound pvc-65010f2c-3fed-4496-92df-a8eaa69cf3b6 25Gi RWO storage-default 35s
elasticsearch-data-quickstart-es-default-1 Bound pvc-030ae9db-ce4f-49cf-b749-00157a45eb6e 25Gi RWO storage-default 34s
elasticsearch-data-quickstart-es-default-2 Bound pvc-cf817916-16f5-4bd1-a957-1262d8397aad 25Gi RWO storage-default 34s
elasticsearch-data-quickstart-es-default-3 Bound pvc-cfb01ece-18f1-4a51-b209-03c10fd66a1f 25Gi RWO storage-default 34s
elasticsearch-data-quickstart-es-default-4 Bound pvc-596e8aa4-2a6e-453a-8b97-02e0daf58d12 25Gi RWO storage-default 34s
elasticsearch-repo-data-quickstart-es-default-0 Bound pvc-bef62d41-1a66-4183-83aa-57e88858619b 50Gi RWX nfs-csi 34s
elasticsearch-repo-data-quickstart-es-default-1 Bound pvc-1498482e-7ab4-4ef8-80ee-e6f7a6d03368 50Gi RWX nfs-csi 34s
elasticsearch-repo-data-quickstart-es-default-2 Bound pvc-2adcf2a3-7786-4f44-aa64-5085f1c586b7 50Gi RWX nfs-csi 34s
elasticsearch-repo-data-quickstart-es-default-3 Bound pvc-6f4bbd3f-48b2-47e4-8561-244e24170a31 50Gi RWX nfs-csi 34s
elasticsearch-repo-data-quickstart-es-default-4 Bound pvc-b3735406-1d74-48d6-a4be-22f62aa48237 50Gi RWX nfs-csi 34s
Any idea what i am doing wrong?
Thank you for feedback and help