External provisioner is not creating volume

Recently deployed a brand new ECK and Elasticsearch Cluster using the official quickstart guide. but my external provisioner which is a NFS server through custom StorageClass is not creating the volume for Elasticsearch Cluster. Let me know if I am doing wrong something. Any help would be appreciated.

waiting for a volume to be created, either by external provisioner "nfs-mavara-du" or manually created by system administrator
kubectl get all -n elastic-system

NAME                                       READY   STATUS    RESTARTS   AGE
pod/elastic-operator-0                     1/1     Running   11         24h
pod/nfs-pod-provisioner-857b88ff47-hd7x9   1/1     Running   0          137m
pod/quickstart-es-default-0                0/1     Pending   0          41m

NAME                              TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/elastic-webhook-server    ClusterIP   10.233.51.69    <none>        443/TCP    25h
service/quickstart-es-default     ClusterIP   None            <none>        9200/TCP   41m
service/quickstart-es-http        ClusterIP   10.233.49.195   <none>        9200/TCP   41m
service/quickstart-es-transport   ClusterIP   None            <none>        9300/TCP   41m

NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/nfs-pod-provisioner   1/1     1            1           137m

NAME                                             DESIRED   CURRENT   READY   AGE
replicaset.apps/nfs-pod-provisioner-857b88ff47   1         1         1       137m

NAME                                     READY   AGE
statefulset.apps/elastic-operator        1/1     25h
statefulset.apps/quickstart-es-default   0/1     41m
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: quickstart
spec:
  version: 7.15.0
  nodeSets:
  - name: default
    count: 1
    config:
      node.store.allow_mmap: false
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 1Gi
        storageClassName: nfs-storageclass2
        volumeMode: Filesystem
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: nfs-storageclass2
parameters:
  archiveOnDelete: "false"
provisioner: nfs-mavara-du
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nfs-pod-provisioner
  labels:
    app: nfs-pod-provisioner
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      app: nfs-pod-provisioner
  template:
    metadata:
      labels:
        app: nfs-pod-provisioner
    spec:
      serviceAccountName: nfs-pod-provisioner-sa # name of service account created in rbac.yaml
      containers:
        - name: nfs-pod-provisioner
          image: quay.io/external_storage/nfs-client-provisioner:latest
          volumeMounts:
            - name: nfs-provisioner-v
              mountPath: /persistentvolumes
          env:
            - name: PROVISIONER_NAME # do not change
              value: nfs-mavara-du # SAME AS PROVISONER NAME VALUE IN STORAGECLASS
            - name: NFS_SERVER # do not change
              value: 10.10.20.10 # Ip of the NFS SERVER
            - name: NFS_PATH # do not change
              value: /dnfs # path to nfs directory setup
      volumes:
       - name: nfs-provisioner-v # same as volumemouts name
         nfs:
           server: 10.10.20.10
           path: /dnfs
kind: ServiceAccount
apiVersion: v1
metadata:
  name: nfs-pod-provisioner-sa
---
kind: ClusterRole # Role of kubernetes
apiVersion: rbac.authorization.k8s.io/v1 # auth API
metadata:
  name: nfs-provisioner-clusterRole
rules:
  - apiGroups: [""] # rules on persistentvolumes
    resources: ["persistentvolumes"]
    verbs: ["get", "list", "watch", "create", "delete"]
  - apiGroups: [""]
    resources: ["persistentvolumeclaims"]
    verbs: ["get", "list", "watch", "update"]
  - apiGroups: ["storage.k8s.io"]
    resources: ["storageclasses"]
    verbs: ["get", "list", "watch"]
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create", "update", "patch"]
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-provisioner-rolebinding
subjects:
  - kind: ServiceAccount
    name: nfs-pod-provisioner-sa # defined on top of file
    namespace: elastic-system
roleRef: # binding cluster role to service account
  kind: ClusterRole
  name: nfs-provisioner-clusterRole # name defined in clusterRole
  apiGroup: rbac.authorization.k8s.io
---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-pod-provisioner-otherRoles
rules:
  - apiGroups: [""]
    resources: ["endpoints"]
    verbs: ["get", "list", "watch", "create", "update", "patch"]
---
kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: nfs-pod-provisioner-otherRoles
subjects:
  - kind: ServiceAccount
    name: nfs-pod-provisioner-sa # same as top of the file
    # replace with namespace where provisioner is deployed
    namespace: elastic-system
roleRef:
  kind: Role
  name: nfs-pod-provisioner-otherRoles
  apiGroup: rbac.authorization.k8s.io

Unfortunately a simple standalone PVC which uses the same StorageClass could not creating the PV through the same external provisioner.

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: nfs-pvc-test
spec:
  storageClassName: nfs-storageclass
  volumeMode: Filesystem
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 50Mi
kubectl -n elastic-system describe pvc/nfs-pvc-test

Name:          nfs-pvc-test
Namespace:     elastic-system
StorageClass:  nfs-storageclass
Status:        Pending
Volume:
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-provisioner: nfs-mavara-du
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
Used By:       <none>
Events:
  Type    Reason                Age                   From                         Message
  ----    ------                ----                  ----                         -------
  Normal  ExternalProvisioning  15s (x12 over 2m39s)  persistentvolume-controller  waiting for a volume to be created, either by external provisioner "nfs-mavara-du" or manually created by system administrator

I can't really help with the NFS storage provisioner and it would probably be good to engage with NFS provisioner project. I noticed however that the nfs-client-provisioner project is deprecated.

There is another more fundamental issue here and that is that it is not recommended to use NFS volumes for the Elasticsearch data volume to begin with. This discussion thread offers a bit more context as to why that is the case and it is also mentioned in the Elasticsearch documentation,

1 Like

Thank you for your answer. I will switch to nfs-subdir-external-provisioner which is currently the best alternative nfs provisioner available.

Could you please let me know what alternative solution do I have for a distributed/shared storage among entire cluster as the NFS can introduces performance issues?

This depends a lot on the environment you are running in. If you are running your cluster on one of the Cloud providers or are using a hosted Kubernetes offering then there are typically already storage provisioners available for example EBS backed volumes on AWS and persistent disks on GCE.

If you are running on premise then there are many options to choose from that it is hard to make a recommendation without knowing a lot more about the context.

You mentioned shared storage, I think it is worth pointing out that using shared storage (as NFS implements it) is not necessary or desirable for Elasticsearch. Each Elasticsearch node needs its own data volume that must not be shared with other nodes. If you need ultimate performance then local storage is also an option. OpenEBS for example offers different data engines for both local and network attached storage.

We have Storage recommendations | Elastic Cloud on Kubernetes [1.8] | Elastic in the ECK documentation with some further guidance.

1 Like