# Init container mount problem in ECK

**URL:** <https://discuss.elastic.co/t/init-container-mount-problem-in-eck/212382>\
**Category:** Elastic Cloud on Kubernetes (ECK)\
**Tags:** docker\
**Created:** [December 18, 2019, 6:26pm UTC](https://discuss.elastic.co/t/init-container-mount-problem-in-eck/212382 "2019-12-18T18:26:59Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![William\_Doran](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/william_doran/32/55467_2.png) [@William\_Doran](https://discuss.elastic.co/u/William_Doran)\
**Post date:** [December 18, 2019, 6:26pm UTC](https://discuss.elastic.co/t/init-container-mount-problem-in-eck/212382/1 "2019-12-18T18:26:59Z")

</div>

HI there, i installed the K8s operator for 1.0.0-beta on my cluster. I attempted to install the default single node cluster using the Quick start guide. My K8s admin has so enforced resource limits so i had to explicitly add resources for my cluster, however the init container fails. Any help would be greatly appreciated. The stackset looks like

```bash
cat <<EOF | zkubectl apply -f -
   apiVersion: elasticsearch.k8s.elastic.co/v1beta1
   kind: Elasticsearch
   metadata:
     name: color-vector
     application: color-vector
   spec:
     version: 7.5.0
     nodeSets:
     - name: default
       count: 1
       config:
         node.master: true
         node.data: true
         node.ingest: true
         node.store.allow_mmap: false
       podTemplate:
         spec:
           initContainers:
           - name: elastic-internal-init-filesystem
             resources:
               requests:
                 memory: 1Gi
                 cpu: 1
               limits:
                 memory: 1Gi
                 cpu: 1
   EOF

```

The pod failed to initialize as the init container failed as below, Im not sure if its a volume mount problem or an incorrect user or permissions:

```bash
>$ kubectl logs color-vector-es-default-0 -c elastic-internal-init-filesystem
chroot: failed to run command '/usr/share/elasticsearch/bin/elasticsearch': No such file or directory

```

The pod and persistent volume claim description are here:

```auto
zkubectl describe pod color-vector-es-default-0
Name: color-vector-es-default-0
Namespace: default
Priority: 0
PriorityClassName: <none>
Node: ip-XXX-XXX-XXX-XXX.eu-central-1.compute.internal/172.31.23.159
Start Time: Wed, 18 Dec 2019 17:48:02 +0000
Labels: common.k8s.elastic.co/type=elasticsearch
                    controller-revision-hash=color-vector-es-default-66b9c4cb6
                    elasticsearch.k8s.elastic.co/cluster-name=color-vector
Status: Pending
Controlled By: StatefulSet/color-vector-es-default
Init Containers:
  elastic-internal-init-filesystem:
    Container ID: docker://d51d56c9d3298260f27cfcf93b48e7ef0932218f34576fb7d086488bc5f00798
    Image: docker.elastic.co/elasticsearch/elasticsearch:7.5.0
    Image ID: docker-pullable://docker.elastic.co/elasticsearch/elasticsearch@sha256:5d4f00b9a5f5f8e882b3301c401ce3bebe1fd09a45b4e70de3761056c76934c3
    Port: <none>
    Host Port: <none>
    State: Terminated
      Reason: Error
      Exit Code: 127
      Started: Wed, 18 Dec 2019 18:09:18 +0000
      Finished: Wed, 18 Dec 2019 18:09:18 +0000
    Last State: Terminated
      Reason: Error
      Exit Code: 127
      Started: Wed, 18 Dec 2019 18:04:14 +0000
      Finished: Wed, 18 Dec 2019 18:04:14 +0000
    Ready: False
    Restart Count: 9
    Limits:
      cpu: 1
      memory: 1Gi
    Requests:
      cpu: 1
      memory: 1Gi
    Environment:
      POD_IP: (v1:status.podIP)
      POD_NAME: color-vector-es-default-0 (v1:metadata.name)
      _PLATFORM_DOCKER_IMAGE: docker.elastic.co/elasticsearch/elasticsearch:7.5.0
    Mounts:
      /mnt/elastic-internal/elasticsearch-config from elastic-internal-elasticsearch-config (ro)
      /mnt/elastic-internal/probe-user from elastic-internal-probe-user (ro)
      /mnt/elastic-internal/scripts from elastic-internal-scripts (ro)
      /mnt/elastic-internal/unicast-hosts from elastic-internal-unicast-hosts (ro)
      /mnt/elastic-internal/xpack-file-realm from elastic-internal-xpack-file-realm (ro)
      /usr/share/elasticsearch/bin from elastic-internal-elasticsearch-bin-local (rw)
      /usr/share/elasticsearch/config from elastic-internal-elasticsearch-config-local (rw)
      /usr/share/elasticsearch/config/http-certs from elastic-internal-http-certificates (ro)
      /usr/share/elasticsearch/config/transport-certs from elastic-internal-transport-certificates (ro)
      /usr/share/elasticsearch/data from elasticsearch-data (rw)
      /usr/share/elasticsearch/logs from elasticsearch-logs (rw)
      /usr/share/elasticsearch/plugins from elastic-internal-elasticsearch-plugins-local (rw)
Containers:
  elasticsearch:
    Container ID:
    Image: docker.elastic.co/elasticsearch/elasticsearch:7.5.0
    Image ID:
    Ports: 9200/TCP, 9300/TCP
    Host Ports: 0/TCP, 0/TCP
    State: Waiting
      Reason: PodInitializing
    Ready: False
    Restart Count: 0
    Limits:
      memory: 2Gi
    Requests:
      cpu: 25m
      memory: 2Gi
    Readiness: exec [bash -c /mnt/elastic-internal/scripts/readiness-probe-script.sh] delay=10s timeout=5s period=5s #success=1 #failure=3
    Environment:
      NSS_SDB_USE_CACHE: no
      POD_IP: (v1:status.podIP)
      POD_NAME: color-vector-es-default-0 (v1:metadata.name)
      PROBE_PASSWORD_FILE: /mnt/elastic-internal/probe-user/elastic-internal-probe
      PROBE_USERNAME: elastic-internal-probe
      READINESS_PROBE_PROTOCOL: https
      _PLATFORM_DOCKER_IMAGE: docker.elastic.co/elasticsearch/elasticsearch:7.5.0
      _PLATFORM_OPENTRACING_TAG_ARTIFACT: docker.elastic.co/elasticsearch/elasticsearch:7.5.0
    Mounts:
      /mnt/elastic-internal/elasticsearch-config from elastic-internal-elasticsearch-config (ro)
      /mnt/elastic-internal/probe-user from elastic-internal-probe-user (ro)
      /mnt/elastic-internal/scripts from elastic-internal-scripts (ro)
      /mnt/elastic-internal/unicast-hosts from elastic-internal-unicast-hosts (ro)
      /mnt/elastic-internal/xpack-file-realm from elastic-internal-xpack-file-realm (ro)
      /usr/share/elasticsearch/bin from elastic-internal-elasticsearch-bin-local (rw)
      /usr/share/elasticsearch/config from elastic-internal-elasticsearch-config-local (rw)
      /usr/share/elasticsearch/config/http-certs from elastic-internal-http-certificates (ro)
      /usr/share/elasticsearch/config/transport-certs from elastic-internal-transport-certificates (ro)
      /usr/share/elasticsearch/data from elasticsearch-data (rw)
      /usr/share/elasticsearch/logs from elasticsearch-logs (rw)
      /usr/share/elasticsearch/plugins from elastic-internal-elasticsearch-plugins-local (rw)
Conditions:
  Type Status
  Initialized False
  Ready False
  ContainersReady False
  PodScheduled True
Volumes:
  elasticsearch-data:
    Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName: elasticsearch-data-color-vector-es-default-0
    ReadOnly: false
  elastic-internal-elasticsearch-bin-local:
    Type: EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  elastic-internal-elasticsearch-config:
    Type: Secret (a volume populated by a Secret)
    SecretName: color-vector-es-default-es-config
    Optional: false
  elastic-internal-elasticsearch-config-local:
    Type: EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  elastic-internal-elasticsearch-plugins-local:
    Type: EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
  elastic-internal-http-certificates:
    Type: Secret (a volume populated by a Secret)
    SecretName: color-vector-es-http-certs-internal
    Optional: false
  elastic-internal-probe-user:
    Type: Secret (a volume populated by a Secret)
    SecretName: color-vector-es-internal-users
    Optional: false
  elastic-internal-scripts:
    Type: ConfigMap (a volume populated by a ConfigMap)
    Name: color-vector-es-scripts
    Optional: false
  elastic-internal-transport-certificates:
    Type: Secret (a volume populated by a Secret)
    SecretName: color-vector-es-transport-certificates
    Optional: false
  elastic-internal-unicast-hosts:
    Type: ConfigMap (a volume populated by a ConfigMap)
    Name: color-vector-es-unicast-hosts
    Optional: false
  elastic-internal-xpack-file-realm:
    Type: Secret (a volume populated by a Secret)
    SecretName: color-vector-es-xpack-file-realm
    Optional: false
  elasticsearch-logs:
    Type: EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s
Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Warning FailedScheduling 21m (x2 over 21m) default-scheduler pod has unbound immediate PersistentVolumeClaims
  Normal Scheduled 21m default-scheduler Successfully assigned default/color-vector-es-default-0 to ip-XXX-XXX-XXX-XXX.eu-central-1.compute.internal
  Normal SuccessfulAttachVolume 21m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-59bd0770-e4a1-487e-b5da-0c98da2a5690"
  Normal Pulled 19m (x5 over 21m) kubelet, ip-XXX-XXX-XXX-XXX.eu-central-1.compute.internal Container image "docker.elastic.co/elasticsearch/elasticsearch:7.5.0" already present on machine
  Normal Created 19m (x5 over 21m) kubelet, ip-XXX-XXX-XXX-XXX.eu-central-1.compute.internal Created container elastic-internal-init-filesystem
  Normal Started 19m (x5 over 21m) kubelet, ip-XXX-XXX-XXX-XXX.eu-central-1.compute.internal Started container elastic-internal-init-filesystem
  Warning BackOff 67s (x95 over 21m) kubelet, ip-XXX-XXX-XXX-XXX.eu-central-1.compute.internal Back-off restarting failed container

```

And the volume claim

````auto
zkubectl describe pvc elasticsearch-data-color-vector-es-default-0

Name: elasticsearch-data-color-vector-es-default-0
Namespace: default
StorageClass: standard
Status: Bound
Volume: pvc-59bd0770-e4a1-487e-b5da-0c98da2a5690
Labels: common.k8s.elastic.co/type=elasticsearch
               elasticsearch.k8s.elastic.co/cluster-name=color-vector
               elasticsearch.k8s.elastic.co/statefulset-name=color-vector-es-default
Annotations: pv.kubernetes.io/bind-completed: yes
               pv.kubernetes.io/bound-by-controller: yes
               volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/aws-ebs
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 1Gi
Access Modes: RWO
VolumeMode: Filesystem
Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Normal ProvisioningSucceeded 22m persistentvolume-controller Successfully provisioned volume pvc-59bd0770-e4a1-487e-b5da-0c98da2a5690 using kubernetes.io/aws-ebs
Mounted By: color-vector-es-default-0```
````

---

<div class="post-metadata">

**Author:** ![charith-elastic](https://avatars.discourse-cdn.com/v4/letter/c/3ec8ea/32.png) [@charith-elastic](https://discuss.elastic.co/u/charith-elastic)\
**Post date:** [December 19, 2019, 11:10am UTC](https://discuss.elastic.co/t/init-container-mount-problem-in-eck/212382/2 "2019-12-19T11:10:30Z")

</div>

Hi @William_Doran

Changing the init container requires some boilerplate. The good news is that the next release of ECK will set resource limits for init containers by default so this particular problem should be solved very soon.

In ECK, if you specify an init container in the pod template with a name that clashes with a built-in init container, we give priority to the user defined container and use it as is. This means that if you want to change something in the init container, you must provide the full definition of the container as the operator does not try to modify it.

To set the resource limits as you wanted, you should provide the following manifest:

```auto
cat <<EOF | kubectl apply -f -
apiVersion: elasticsearch.k8s.elastic.co/v1beta1
kind: Elasticsearch
metadata:
 name: color-vector
 application: color-vector
spec:
 version: 7.5.0
 nodeSets:
 - name: default
   count: 1
   config:
     node.master: true
     node.data: true
     node.ingest: true
     node.store.allow_mmap: false
   podTemplate:
     spec:
       initContainers:
       - name: elastic-internal-init-filesystem
         image: docker.elastic.co/elasticsearch/elasticsearch:7.5.0
         imagePullPolicy: IfNotPresent
         command:
          - bash
          - -c
          - /mnt/elastic-internal/scripts/prepare-fs.sh
         resources:
           requests:
             memory: 1Gi
             cpu: 1
           limits:
             memory: 1Gi
             cpu: 1
         volumeMounts:
          - mountPath: /mnt/elastic-internal/elasticsearch-config-local
            name: elastic-internal-elasticsearch-config-local
          - mountPath: /mnt/elastic-internal/elasticsearch-plugins-local
            name: elastic-internal-elasticsearch-plugins-local
          - mountPath: /mnt/elastic-internal/elasticsearch-bin-local
            name: elastic-internal-elasticsearch-bin-local
          - mountPath: /mnt/elastic-internal/transport-certificates
            name: elastic-internal-transport-certificates
            readOnly: true
          - mountPath: /mnt/elastic-internal/scripts
            name: elastic-internal-scripts
            readOnly: true
          - mountPath: /usr/share/elasticsearch/data
            name: elasticsearch-data
          - mountPath: /usr/share/elasticsearch/logs
            name: elasticsearch-logs
          - mountPath: /mnt/elastic-internal/downward-api
            name: downward-api
            readOnly: true
          - mountPath: /mnt/elastic-internal/elasticsearch-config
            name: elastic-internal-elasticsearch-config
            readOnly: true
          - mountPath: /usr/share/elasticsearch/config/http-certs
            name: elastic-internal-http-certificates
            readOnly: true
          - mountPath: /mnt/elastic-internal/probe-user
            name: elastic-internal-probe-user
            readOnly: true
          - mountPath: /mnt/elastic-internal/unicast-hosts
            name: elastic-internal-unicast-hosts
            readOnly: true
          - mountPath: /mnt/elastic-internal/xpack-file-realm
            name: elastic-internal-xpack-file-realm
            readOnly: true
EOF

```

---

<div class="post-metadata">

**Author:** ![William\_Doran](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/william_doran/32/55467_2.png) [@William\_Doran](https://discuss.elastic.co/u/William_Doran)\
**Post date:** [December 20, 2019, 8:59am UTC](https://discuss.elastic.co/t/init-container-mount-problem-in-eck/212382/3 "2019-12-20T08:59:08Z")

</div>

Thanks very much for the speedy response...ran into some issues with one of the mounts, but removed it and everything seems to be working fine so far!!

````auto
            name: downward-api```
````

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 4, 2022, 7:37am UTC](https://discuss.elastic.co/t/init-container-mount-problem-in-eck/212382/4 "2022-11-04T07:37:58Z")

</div>