Init container mount problem in ECK

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

HI there, i installed the K8s operator for 1.0.0-beta on my cluster. I attempted to install the default single node cluster using the Quick start guide. My K8s admin has so enforced resource limits so i had to explicitly add resources for my cluster, however the init container fails. Any help would be greatly appreciated. The stackset looks like

cat <<EOF | zkubectl apply -f -
   apiVersion: elasticsearch.k8s.elastic.co/v1beta1
   kind: Elasticsearch
   metadata:
     name: color-vector
     application: color-vector
   spec:
     version: 7.5.0
     nodeSets:
     - name: default
       count: 1
       config:
         node.master: true
         node.data: true
         node.ingest: true
         node.store.allow_mmap: false
       podTemplate:
         spec:
           initContainers:
           - name: elastic-internal-init-filesystem
             resources:
               requests:
                 memory: 1Gi
                 cpu: 1
               limits:
                 memory: 1Gi
                 cpu: 1
   EOF

The pod failed to initialize as the init container failed as below, Im not sure if its a volume mount problem or an incorrect user or permissions:

>$ kubectl logs color-vector-es-default-0 -c elastic-internal-init-filesystem
chroot: failed to run command '/usr/share/elasticsearch/bin/elasticsearch': No such file or directory

The pod and persistent volume claim description are here:

zkubectl describe pod color-vector-es-default-0
Name:               color-vector-es-default-0
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               ip-XXX-XXX-XXX-XXX.eu-central-1.compute.internal/172.31.23.159
Start Time:         Wed, 18 Dec 2019 17:48:02 +0000
Labels:             common.k8s.elastic.co/type=elasticsearch
                    controller-revision-hash=color-vector-es-default-66b9c4cb6
                    elasticsearch.k8s.elastic.co/cluster-name=color-vector
Status:             Pending
Controlled By:      StatefulSet/color-vector-es-default
Init Containers:
  elastic-internal-init-filesystem:
    Container ID:   docker://d51d56c9d3298260f27cfcf93b48e7ef0932218f34576fb7d086488bc5f00798
    Image:          docker.elastic.co/elasticsearch/elasticsearch:7.5.0
    Image ID:       docker-pullable://docker.elastic.co/elasticsearch/elasticsearch@sha256:5d4f00b9a5f5f8e882b3301c401ce3bebe1fd09a45b4e70de3761056c76934c3
    Port:           <none>
    Host Port:      <none>
    State:          Terminated
      Reason:       Error
      Exit Code:    127
      Started:      Wed, 18 Dec 2019 18:09:18 +0000
      Finished:     Wed, 18 Dec 2019 18:09:18 +0000
    Last State:     Terminated
      Reason:       Error
      Exit Code:    127
      Started:      Wed, 18 Dec 2019 18:04:14 +0000
      Finished:     Wed, 18 Dec 2019 18:04:14 +0000
    Ready:          False
    Restart Count:  9
    Limits:
      cpu:     1
      memory:  1Gi
    Requests:
      cpu:     1
      memory:  1Gi
    Environment:
      POD_IP:                                           (v1:status.podIP)
      POD_NAME:                                        color-vector-es-default-0 (v1:metadata.name)
      _PLATFORM_DOCKER_IMAGE:                          docker.elastic.co/elasticsearch/elasticsearch:7.5.0
    Mounts:
      /mnt/elastic-internal/elasticsearch-config from elastic-internal-elasticsearch-config (ro)
      /mnt/elastic-internal/probe-user from elastic-internal-probe-user (ro)
      /mnt/elastic-internal/scripts from elastic-internal-scripts (ro)
      /mnt/elastic-internal/unicast-hosts from elastic-internal-unicast-hosts (ro)
      /mnt/elastic-internal/xpack-file-realm from elastic-internal-xpack-file-realm (ro)
      /usr/share/elasticsearch/bin from elastic-internal-elasticsearch-bin-local (rw)
      /usr/share/elasticsearch/config from elastic-internal-elasticsearch-config-local (rw)
      /usr/share/elasticsearch/config/http-certs from elastic-internal-http-certificates (ro)
      /usr/share/elasticsearch/config/transport-certs from elastic-internal-transport-certificates (ro)
      /usr/share/elasticsearch/data from elasticsearch-data (rw)
      /usr/share/elasticsearch/logs from elasticsearch-logs (rw)
      /usr/share/elasticsearch/plugins from elastic-internal-elasticsearch-plugins-local (rw)
Containers:
  elasticsearch:
    Container ID:
    Image:          docker.elastic.co/elasticsearch/elasticsearch:7.5.0
    Image ID:
    Ports:          9200/TCP, 9300/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Waiting
      Reason:       PodInitializing
    Ready:          False
    Restart Count:  0
    Limits:
      memory:  2Gi
    Requests:
      cpu:      25m
      memory:   2Gi
    Readiness:  exec [bash -c /mnt/elastic-internal/scripts/readiness-probe-script.sh] delay=10s timeout=5s period=5s #success=1 #failure=3
    Environment:
      NSS_SDB_USE_CACHE:                               no
      POD_IP:                                           (v1:status.podIP)
      POD_NAME:                                        color-vector-es-default-0 (v1:metadata.name)
      PROBE_PASSWORD_FILE:                             /mnt/elastic-internal/probe-user/elastic-internal-probe
      PROBE_USERNAME:                                  elastic-internal-probe
      READINESS_PROBE_PROTOCOL:                        https
      _PLATFORM_DOCKER_IMAGE:                          docker.elastic.co/elasticsearch/elasticsearch:7.5.0
      _PLATFORM_OPENTRACING_TAG_ARTIFACT:              docker.elastic.co/elasticsearch/elasticsearch:7.5.0
    Mounts:
      /mnt/elastic-internal/elasticsearch-config from elastic-internal-elasticsearch-config (ro)
      /mnt/elastic-internal/probe-user from elastic-internal-probe-user (ro)
      /mnt/elastic-internal/scripts from elastic-internal-scripts (ro)
      /mnt/elastic-internal/unicast-hosts from elastic-internal-unicast-hosts (ro)
      /mnt/elastic-internal/xpack-file-realm from elastic-internal-xpack-file-realm (ro)
      /usr/share/elasticsearch/bin from elastic-internal-elasticsearch-bin-local (rw)
      /usr/share/elasticsearch/config from elastic-internal-elasticsearch-config-local (rw)
      /usr/share/elasticsearch/config/http-certs from elastic-internal-http-certificates (ro)
      /usr/share/elasticsearch/config/transport-certs from elastic-internal-transport-certificates (ro)
      /usr/share/elasticsearch/data from elasticsearch-data (rw)
      /usr/share/elasticsearch/logs from elasticsearch-logs (rw)
      /usr/share/elasticsearch/plugins from elastic-internal-elasticsearch-plugins-local (rw)
Conditions:
  Type              Status
  Initialized       False
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  elasticsearch-data:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  elasticsearch-data-color-vector-es-default-0
    ReadOnly:   false
  elastic-internal-elasticsearch-bin-local:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  elastic-internal-elasticsearch-config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  color-vector-es-default-es-config
    Optional:    false
  elastic-internal-elasticsearch-config-local:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  elastic-internal-elasticsearch-plugins-local:
    Type:    EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  elastic-internal-http-certificates:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  color-vector-es-http-certs-internal
    Optional:    false
  elastic-internal-probe-user:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  color-vector-es-internal-users
    Optional:    false
  elastic-internal-scripts:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      color-vector-es-scripts
    Optional:  false
  elastic-internal-transport-certificates:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  color-vector-es-transport-certificates
    Optional:    false
  elastic-internal-unicast-hosts:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      color-vector-es-unicast-hosts
    Optional:  false
  elastic-internal-xpack-file-realm:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  color-vector-es-xpack-file-realm
    Optional:    false
  elasticsearch-logs:
    Type:        EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type     Reason                  Age                 From                                                     Message
  ----     ------                  ----                ----                                                     -------
  Warning  FailedScheduling        21m (x2 over 21m)   default-scheduler                                        pod has unbound immediate PersistentVolumeClaims
  Normal   Scheduled               21m                 default-scheduler                                        Successfully assigned default/color-vector-es-default-0 to ip-XXX-XXX-XXX-XXX.eu-central-1.compute.internal
  Normal   SuccessfulAttachVolume  21m                 attachdetach-controller                                  AttachVolume.Attach succeeded for volume "pvc-59bd0770-e4a1-487e-b5da-0c98da2a5690"
  Normal   Pulled                  19m (x5 over 21m)   kubelet, ip-XXX-XXX-XXX-XXX.eu-central-1.compute.internal  Container image "docker.elastic.co/elasticsearch/elasticsearch:7.5.0" already present on machine
  Normal   Created                 19m (x5 over 21m)   kubelet, ip-XXX-XXX-XXX-XXX.eu-central-1.compute.internal  Created container elastic-internal-init-filesystem
  Normal   Started                 19m (x5 over 21m)   kubelet, ip-XXX-XXX-XXX-XXX.eu-central-1.compute.internal  Started container elastic-internal-init-filesystem
  Warning  BackOff                 67s (x95 over 21m)  kubelet, ip-XXX-XXX-XXX-XXX.eu-central-1.compute.internal  Back-off restarting failed container

And the volume claim

zkubectl describe pvc elasticsearch-data-color-vector-es-default-0

Name:          elasticsearch-data-color-vector-es-default-0
Namespace:     default
StorageClass:  standard
Status:        Bound
Volume:        pvc-59bd0770-e4a1-487e-b5da-0c98da2a5690
Labels:        common.k8s.elastic.co/type=elasticsearch
               elasticsearch.k8s.elastic.co/cluster-name=color-vector
               elasticsearch.k8s.elastic.co/statefulset-name=color-vector-es-default
Annotations:   pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/aws-ebs
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:      1Gi
Access Modes:  RWO
VolumeMode:    Filesystem
Events:
  Type       Reason                 Age   From                         Message
  ----       ------                 ----  ----                         -------
  Normal     ProvisioningSucceeded  22m   persistentvolume-controller  Successfully provisioned volume pvc-59bd0770-e4a1-487e-b5da-0c98da2a5690 using kubernetes.io/aws-ebs
Mounted By:  color-vector-es-default-0```
2

Hi @William_Doran

Changing the init container requires some boilerplate. The good news is that the next release of ECK will set resource limits for init containers by default so this particular problem should be solved very soon.

In ECK, if you specify an init container in the pod template with a name that clashes with a built-in init container, we give priority to the user defined container and use it as is. This means that if you want to change something in the init container, you must provide the full definition of the container as the operator does not try to modify it.

To set the resource limits as you wanted, you should provide the following manifest:

cat <<EOF | kubectl apply -f -
apiVersion: elasticsearch.k8s.elastic.co/v1beta1
kind: Elasticsearch
metadata:
 name: color-vector
 application: color-vector
spec:
 version: 7.5.0
 nodeSets:
 - name: default
   count: 1
   config:
     node.master: true
     node.data: true
     node.ingest: true
     node.store.allow_mmap: false
   podTemplate:
     spec:
       initContainers:
       - name: elastic-internal-init-filesystem
         image: docker.elastic.co/elasticsearch/elasticsearch:7.5.0
         imagePullPolicy: IfNotPresent
         command:
          - bash
          - -c
          - /mnt/elastic-internal/scripts/prepare-fs.sh
         resources:
           requests:
             memory: 1Gi
             cpu: 1
           limits:
             memory: 1Gi
             cpu: 1
         volumeMounts:
          - mountPath: /mnt/elastic-internal/elasticsearch-config-local
            name: elastic-internal-elasticsearch-config-local
          - mountPath: /mnt/elastic-internal/elasticsearch-plugins-local
            name: elastic-internal-elasticsearch-plugins-local
          - mountPath: /mnt/elastic-internal/elasticsearch-bin-local
            name: elastic-internal-elasticsearch-bin-local
          - mountPath: /mnt/elastic-internal/transport-certificates
            name: elastic-internal-transport-certificates
            readOnly: true
          - mountPath: /mnt/elastic-internal/scripts
            name: elastic-internal-scripts
            readOnly: true
          - mountPath: /usr/share/elasticsearch/data
            name: elasticsearch-data
          - mountPath: /usr/share/elasticsearch/logs
            name: elasticsearch-logs
          - mountPath: /mnt/elastic-internal/downward-api
            name: downward-api
            readOnly: true
          - mountPath: /mnt/elastic-internal/elasticsearch-config
            name: elastic-internal-elasticsearch-config
            readOnly: true
          - mountPath: /usr/share/elasticsearch/config/http-certs
            name: elastic-internal-http-certificates
            readOnly: true
          - mountPath: /mnt/elastic-internal/probe-user
            name: elastic-internal-probe-user
            readOnly: true
          - mountPath: /mnt/elastic-internal/unicast-hosts
            name: elastic-internal-unicast-hosts
            readOnly: true
          - mountPath: /mnt/elastic-internal/xpack-file-realm
            name: elastic-internal-xpack-file-realm
            readOnly: true
EOF
3

Thanks very much for the speedy response...ran into some issues with one of the mounts, but removed it and everything seems to be working fine so far!!

            name: downward-api```