ElasticSearch statefulset not using data volume

[yeyus (Jesus Trujillo)]

Hi fellow users,

I've deployed the following single-node setup in my k8s cluster:
eck operator: 1.5.0
elasticsearch: 7.12.0

with the following config:

    apiVersion: elasticsearch.k8s.elastic.co/v1
    kind: Elasticsearch
    metadata:
      name: sample
      namespace: sample
    spec:
      version: 7.12.0
      nodeSets:
      - name: default
        count: 1
        config:
          node.store.allow_mmap: false
          node.master: true
          node.ingest: true
          node.data: true
        volumeClaimTemplates:
          - metadata:
              name: sample-elastic-data
            spec:
              accessModes:
              - ReadWriteOnce
              storageClassName: nfs-client
              resources:
                requests:
                  storage: 50Gi
      http:
        service:
          spec:
            type: LoadBalancer

After that everything seemed to work fine, a PVC and PV where created and bound with the proper resource requirements. I could see the provisioner created the appropriate folders and stuff.

But when checking the definition for the created pod I discovered that the elasticsearch pod is not using the created PV and PVC for storing data. It's defined in the volumes section but not in volumeMounts. The volume is therefore not used for storing the data.

    volumeMounts:
        - mountPath: /mnt/elastic-internal/downward-api
          name: downward-api
          readOnly: true
        - mountPath: /usr/share/elasticsearch/bin
          name: elastic-internal-elasticsearch-bin-local
        - mountPath: /mnt/elastic-internal/elasticsearch-config
          name: elastic-internal-elasticsearch-config
          readOnly: true
        - mountPath: /usr/share/elasticsearch/config
          name: elastic-internal-elasticsearch-config-local
        - mountPath: /usr/share/elasticsearch/plugins
          name: elastic-internal-elasticsearch-plugins-local
        - mountPath: /usr/share/elasticsearch/config/http-certs
          name: elastic-internal-http-certificates
          readOnly: true
        - mountPath: /mnt/elastic-internal/probe-user
          name: elastic-internal-probe-user
          readOnly: true
        - mountPath: /usr/share/elasticsearch/config/transport-remote-certs/
          name: elastic-internal-remote-certificate-authorities
          readOnly: true
        - mountPath: /mnt/elastic-internal/scripts
          name: elastic-internal-scripts
          readOnly: true
        - mountPath: /usr/share/elasticsearch/config/transport-certs
          name: elastic-internal-transport-certificates
          readOnly: true
        - mountPath: /mnt/elastic-internal/unicast-hosts
          name: elastic-internal-unicast-hosts
          readOnly: true
        - mountPath: /mnt/elastic-internal/xpack-file-realm
          name: elastic-internal-xpack-file-realm
          readOnly: true
        - mountPath: /usr/share/elasticsearch/logs
          name: elasticsearch-logs

    volumes:
      - name: sample-elastic-data
        persistentVolumeClaim:
          claimName: sample-elastic-data-sample-es-default-0
      - downwardAPI:
          defaultMode: 420
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.labels
            path: labels
        name: downward-api
      - emptyDir: {}
        name: elastic-internal-elasticsearch-bin-local
      - name: elastic-internal-elasticsearch-config
        secret:
          defaultMode: 420
          optional: false
          secretName: sample-es-default-es-config
      - emptyDir: {}
        name: elastic-internal-elasticsearch-config-local
      - emptyDir: {}
        name: elastic-internal-elasticsearch-plugins-local
      - name: elastic-internal-http-certificates
        secret:
          defaultMode: 420
          optional: false
          secretName: sample-es-http-certs-internal
      - name: elastic-internal-probe-user
        secret:
          defaultMode: 420
          items:
          - key: elastic-internal-probe
            path: elastic-internal-probe
          optional: false
          secretName: sample-es-internal-users
      - name: elastic-internal-remote-certificate-authorities
        secret:
          defaultMode: 420
          optional: false
          secretName: sample-es-remote-ca
      - configMap:
          defaultMode: 493
          name: sample-es-scripts
          optional: false
        name: elastic-internal-scripts
      - name: elastic-internal-transport-certificates
        secret:
          defaultMode: 420
          optional: false
          secretName: sample-es-default-es-transport-certs
      - configMap:
          defaultMode: 420
          name: sample-es-unicast-hosts
          optional: false
        name: elastic-internal-unicast-hosts
      - name: elastic-internal-xpack-file-realm
        secret:
          defaultMode: 420
          optional: false
          secretName: sample-es-xpack-file-realm
      - emptyDir: {}
        name: elasticsearch-logs

As you can see sample-elastic-data is not referenced in volumeMounts and therefore the PV is not used for storage. I suspect the data is just keep as volatile pod state that will disappear once the pod is killed for any reason.

Is this a bug on eck-operator or in my configuration file?.

Thanks.

[dkow (David Kowalski)]

Hi @yeyus, thanks for your question.

The name of the volume claim template has to be elasticsearch-data (instead of sample-elastic-data that you defined). Having a well-known name allows to communicate to the operator whether this template should be used for Elasticsearch data specifically or it's used by the user for another purpose. You can take a look at our documentation if you want to know more details.

Please let us know if you have any other questions.

Thanks,
David