Quickstart: tolerations issues

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
#1

Hi!

I have a 3-node Kubernetes cluster running with OpenStack VM (Ubuntu 20). I wanted to create an ElasticSearch Cluster but I have to say that I am a bit confused with the instructions, especially about setting up the PV, PC and StorageClass.

I have now a toleration issue on my pod:

# kc describe pod
...
Events:
  Type     Reason            Age                  From               Message
  ----     ------            ----                 ----               -------
  Warning  FailedScheduling  59s (x7 over 7m39s)  default-scheduler  0/3 nodes are available: 1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, 2 node(s) didn't find available persistent volumes to bind.

What could be the problem?

More information:

k8s-controller eck # kc get es
NAME         HEALTH    NODES   VERSION   PHASE             AGE
quickstart   unknown           7.6.1     ApplyingChanges   12m
k8s-controller eck # kc get pv
NAME         CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS      CLAIM                                         STORAGECLASS    REASON   AGE
es-storage   1Gi        RWO            Retain           Available   /elasticsearch-data-quickstart-es-default-0   local-storage            15m
k8s-controller eck # kc get pvc
NAME                                         STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS    AGE
elasticsearch-data-quickstart-es-default-0   Pending                                      local-storage   11m
k8s-controller eck # kc get storageclass
NAME            PROVISIONER                    RECLAIMPOLICY   VOLUMEBINDINGMODE      ALLOWVOLUMEEXPANSION   AGE
local-storage   kubernetes.io/no-provisioner   Delete          WaitForFirstConsumer   false                  14m

Thank you for your help!

#2

It looks like you have an available PV, but the PVC stays in a Pending status. Why can't it bind to the available PV? Can you inspect the PVC resource, it may give us more details? kubectl describe pvc elasticsearch-data-quickstart-es-default-0

#3 
k8s-controller ~ # kubectl describe pvc elasticsearch-data-quickstart-es-default-0 
Name:          elasticsearch-data-quickstart-es-default-0
Namespace:     default
StorageClass:  local-storage
Status:        Pending
Volume:        
Labels:        common.k8s.elastic.co/type=elasticsearch
               elasticsearch.k8s.elastic.co/cluster-name=quickstart
               elasticsearch.k8s.elastic.co/statefulset-name=quickstart-es-default
Annotations:   <none>
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      
Access Modes:  
VolumeMode:    Filesystem
Used By:       quickstart-es-default-0
Events:
  Type    Reason               Age                        From                         Message
  ----    ------               ----                       ----                         -------
  Normal  WaitForPodScheduled  4m58s (x26324 over 4d13h)  persistentvolume-controller  waiting for pod quickstart-es-default-0 to be scheduled
#4

or maybe there is a way to do this without PV and PVC at all? With local storage, maybe?

#5

We really recommend using PVs and PVCs, which work well with local storage.
In your case it looks like you have one PV created, but the Pod can't be scheduled on 2 out of 3 k8s nodes. Your PV looks like a local volume, is there a chance it happens to be located on the 1 k8s node that has the incompatible taint?

1 node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate

#6

The PV's taint configuration is such that it can be only on one of the worker nodes:

apiVersion: v1
kind: PersistentVolume
metadata:
  creationTimestamp: "2021-03-11T17:14:20Z"
  finalizers:
  - kubernetes.io/pv-protection
  name: es-storage
  resourceVersion: "1302627"
  uid: 2725d615-103f-46d0-918a-f70588965e2f
spec:
  accessModes:
  - ReadWriteOnce
  capacity:
    storage: 1Gi
  claimRef:
    name: elasticsearch-data-quickstart-es-default-0
  local:
    path: /mnt/data
  nodeAffinity:
    required:
      nodeSelectorTerms:
      - matchExpressions:
        - key: kubernetes.io/hostname
          operator: In
          values:
          - k8s-worker-1
          - k8s-worker-2
  persistentVolumeReclaimPolicy: Retain
  storageClassName: local-storage
  volumeMode: Filesystem
status:
  phase: Available