# ECK : can't have green elasticsearch cluster

**URL:** <https://discuss.elastic.co/t/eck-cant-have-green-elasticsearch-cluster/354106>\
**Category:** Elasticsearch\
**Tags:** docker\
**Created:** [February 26, 2024, 9:55am UTC](https://discuss.elastic.co/t/eck-cant-have-green-elasticsearch-cluster/354106 "2024-02-26T09:55:39Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Bobflyer](https://avatars.discourse-cdn.com/v4/letter/b/67e7ee/32.png) [@Bobflyer](https://discuss.elastic.co/u/Bobflyer)\
**Post date:** [February 26, 2024, 9:55am UTC](https://discuss.elastic.co/t/eck-cant-have-green-elasticsearch-cluster/354106/1 "2024-02-26T09:55:40Z")

</div>

Hello,

I try to deploy an elasticsearch cluster on my docker-desktop kubernetes test cluster. The final goal is to deploy it in a k3s cluster on my raspberry pies.

Here my kubernetes manifest :

```auto
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch-cluster
  namespace: monitoring
spec:
  version: 8.12.2
  # pour avancer
  # https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-node-configuration.html
  nodeSets:
  - name: master
    count: 1
    config:
      node.roles: ["master"]
      xpack.security.enabled: false
      xpack.security.http.ssl.enabled: false
      xpack.security.transport.ssl.enabled: false
    # volumeClaimTemplates:
    # - metadata:
    # name: elasticsearch-data
    # namespace: monitoring
    # spec:
    # accessModes:
    # - ReadWriteOnce
    # resources:
    # requests:
    # storage: 50Gi
    # storageClassName: elasticsearch-master-data
    podTemplate:
      spec:
        containers:
          - name: elasticsearch-master
            image: elasticsearch:8.12.2
            readinessProbe:
              initialDelaySeconds: 60
              exec:
                command:
                - bash
                - -c
                - /mnt/elastic-internal/scripts/readiness-probe-script.sh
            env:
            - name: READINESS_PROBE_TIMEOUT
              value: "60"
            # volumeMounts:
            # - name: elasticsearch-data
            # mountPath: /usr/share/elasticsearch/data
            resources:
              requests:
                memory: 4Gi
                cpu: 1
              limits:
                memory: 4Gi
                cpu: 4
  - name: data
    count: 1
    config:
      node.roles: ["data", "ingest", "ml", "transform"]
      xpack.security.enabled: false
      xpack.security.http.ssl.enabled: false
      xpack.security.transport.ssl.enabled: false
    # volumeClaimTemplates:
    # - metadata:
    # name: elasticsearch-data
    # namespace: monitoring
    # spec:
    # accessModes:
    # - ReadWriteOnce
    # resources:
    # requests:
    # storage: 50Gi
    # storageClassName: elasticsearch-client-data
    podTemplate:
      spec:
        containers:
          - name: elasticsearch-client
            image: elasticsearch:8.12.2
            readinessProbe:
              initialDelaySeconds: 60
              exec: 
                command:
                - bash
                - -c
                - /mnt/elastic-internal/scripts/readiness-probe-script.sh
            env:
            - name: READINESS_PROBE_TIMEOUT
              value: "60"
            # volumeMounts:
            # - name: elasticsearch-data
            # mountPath: /usr/share/elasticsearch/data
            resources:
              requests:
                memory: 4Gi
                cpu: 1
              limits:
                memory: 4Gi
                cpu: 4

```

When I describe the master pod :

```auto
kubectl describe -n monitoring pod elasticsearch-cluster-es-master-0
Name: elasticsearch-cluster-es-master-0
Namespace: monitoring
Priority: 0
Service Account: default
Node: docker-desktop/192.168.65.3
Start Time: Mon, 26 Feb 2024 10:49:38 +0100
Labels: apps.kubernetes.io/pod-index=0
                  common.k8s.elastic.co/type=elasticsearch
                  controller-revision-hash=elasticsearch-cluster-es-master-76d967c889
                  elasticsearch.k8s.elastic.co/cluster-name=elasticsearch-cluster
                  elasticsearch.k8s.elastic.co/http-scheme=https
                  elasticsearch.k8s.elastic.co/node-data=false
                  elasticsearch.k8s.elastic.co/node-data_cold=false
                  elasticsearch.k8s.elastic.co/node-data_content=false
                  elasticsearch.k8s.elastic.co/node-data_frozen=false
                  elasticsearch.k8s.elastic.co/node-data_hot=false
                  elasticsearch.k8s.elastic.co/node-data_warm=false
                  elasticsearch.k8s.elastic.co/node-ingest=false
                  elasticsearch.k8s.elastic.co/node-master=true
                  elasticsearch.k8s.elastic.co/node-ml=false
                  elasticsearch.k8s.elastic.co/node-remote_cluster_client=false
                  elasticsearch.k8s.elastic.co/node-transform=false
                  elasticsearch.k8s.elastic.co/node-voting_only=false
                  elasticsearch.k8s.elastic.co/statefulset-name=elasticsearch-cluster-es-master
                  elasticsearch.k8s.elastic.co/version=8.12.2
                  statefulset.kubernetes.io/pod-name=elasticsearch-cluster-es-master-0
Annotations: co.elastic.logs/module: elasticsearch
                  elasticsearch.k8s.elastic.co/config-hash: 1377360743
                  policy.k8s.elastic.co/elasticsearch-config-mounts-hash:
                  update.k8s.elastic.co/timestamp: 2024-02-26T09:49:39.681958207Z
Status: Running
IP: 10.1.1.39
IPs:
  IP: 10.1.1.39
Controlled By: StatefulSet/elasticsearch-cluster-es-master
Init Containers:
  elastic-internal-init-filesystem:
    Container ID: docker://302e25d7fe47d93262e5b14171af00fd668e3f89d91f40ec8e29f4f9367b322b
    Image: docker.elastic.co/elasticsearch/elasticsearch:8.12.2
    Image ID: docker-pullable://docker.elastic.co/elasticsearch/elasticsearch@sha256:81b3bbd42d4d5e238e7ee31b7e99acab0bac1fca4eb2fa88415bad5482e52b88
    Port: <none>
    Host Port: <none>
    Command:
      bash
      -c
      /mnt/elastic-internal/scripts/prepare-fs.sh
    State: Terminated
      Reason: Completed
      Exit Code: 0
      Started: Mon, 26 Feb 2024 10:49:38 +0100
      Finished: Mon, 26 Feb 2024 10:49:40 +0100
    Ready: True
    Restart Count: 0
    Limits:
      cpu: 100m
      memory: 50Mi
    Requests:
      cpu: 100m
      memory: 50Mi
    Environment:
      POD_IP: (v1:status.podIP)
      POD_NAME: elasticsearch-cluster-es-master-0 (v1:metadata.name)
      NODE_NAME: (v1:spec.nodeName)
      NAMESPACE: monitoring (v1:metadata.namespace)
      PROBE_PASSWORD_PATH: /mnt/elastic-internal/pod-mounted-users/elastic-internal-probe
      PROBE_USERNAME: elastic-internal-probe
      READINESS_PROBE_PROTOCOL: https
      HEADLESS_SERVICE_NAME: elasticsearch-cluster-es-master
      NSS_SDB_USE_CACHE: no
    Mounts:
      /mnt/elastic-internal/downward-api from downward-api (ro)
      /mnt/elastic-internal/elasticsearch-bin-local from elastic-internal-elasticsearch-bin-local (rw)
      /mnt/elastic-internal/elasticsearch-config from elastic-internal-elasticsearch-config (ro)
      /mnt/elastic-internal/elasticsearch-config-local from elastic-internal-elasticsearch-config-local (rw)
      /mnt/elastic-internal/elasticsearch-plugins-local from elastic-internal-elasticsearch-plugins-local (rw)
      /mnt/elastic-internal/pod-mounted-users from elastic-internal-probe-user (ro)
      /mnt/elastic-internal/scripts from elastic-internal-scripts (ro)
      /mnt/elastic-internal/transport-certificates from elastic-internal-transport-certificates (ro)
      /mnt/elastic-internal/unicast-hosts from elastic-internal-unicast-hosts (ro)
      /mnt/elastic-internal/xpack-file-realm from elastic-internal-xpack-file-realm (ro)
      /tmp from tmp-volume (rw)
      /usr/share/elasticsearch/config/http-certs from elastic-internal-http-certificates (ro)
      /usr/share/elasticsearch/config/operator from file-settings (ro)
      /usr/share/elasticsearch/config/transport-remote-certs/ from elastic-internal-remote-certificate-authorities (ro)
      /usr/share/elasticsearch/data from elasticsearch-data (rw)
      /usr/share/elasticsearch/logs from elasticsearch-logs (rw)
  elastic-internal-suspend:
    Container ID: docker://5a88f4aa4dd11540da7ecdcf9b07bf78e4233750ca09ef0ba63eed2d9d7d099c
    Image: docker.elastic.co/elasticsearch/elasticsearch:8.12.2
    Image ID: docker-pullable://docker.elastic.co/elasticsearch/elasticsearch@sha256:81b3bbd42d4d5e238e7ee31b7e99acab0bac1fca4eb2fa88415bad5482e52b88
    Port: <none>
    Host Port: <none>
    Command:
      bash
      -c
      /mnt/elastic-internal/scripts/suspend.sh
    State: Terminated
      Reason: Completed
      Exit Code: 0
      Started: Mon, 26 Feb 2024 10:49:41 +0100
      Finished: Mon, 26 Feb 2024 10:49:41 +0100
    Ready: True
    Restart Count: 0
    Limits:
      memory: 2Gi
    Requests:
      memory: 2Gi
    Environment:
      POD_IP: (v1:status.podIP)
      POD_NAME: elasticsearch-cluster-es-master-0 (v1:metadata.name)
      NODE_NAME: (v1:spec.nodeName)
      NAMESPACE: monitoring (v1:metadata.namespace)
      PROBE_PASSWORD_PATH: /mnt/elastic-internal/pod-mounted-users/elastic-internal-probe
      PROBE_USERNAME: elastic-internal-probe
      READINESS_PROBE_PROTOCOL: https
      HEADLESS_SERVICE_NAME: elasticsearch-cluster-es-master
      NSS_SDB_USE_CACHE: no
    Mounts:
      /mnt/elastic-internal/downward-api from downward-api (ro)
      /mnt/elastic-internal/elasticsearch-config from elastic-internal-elasticsearch-config (ro)
      /mnt/elastic-internal/pod-mounted-users from elastic-internal-probe-user (ro)
      /mnt/elastic-internal/scripts from elastic-internal-scripts (ro)
      /mnt/elastic-internal/unicast-hosts from elastic-internal-unicast-hosts (ro)
      /mnt/elastic-internal/xpack-file-realm from elastic-internal-xpack-file-realm (ro)
      /tmp from tmp-volume (rw)
      /usr/share/elasticsearch/bin from elastic-internal-elasticsearch-bin-local (rw)
      /usr/share/elasticsearch/config from elastic-internal-elasticsearch-config-local (rw)
      /usr/share/elasticsearch/config/http-certs from elastic-internal-http-certificates (ro)
      /usr/share/elasticsearch/config/operator from file-settings (ro)
      /usr/share/elasticsearch/config/transport-certs from elastic-internal-transport-certificates (ro)
      /usr/share/elasticsearch/config/transport-remote-certs/ from elastic-internal-remote-certificate-authorities (ro)
      /usr/share/elasticsearch/data from elasticsearch-data (rw)
      /usr/share/elasticsearch/logs from elasticsearch-logs (rw)
      /usr/share/elasticsearch/plugins from elastic-internal-elasticsearch-plugins-local (rw)
Containers:
  elasticsearch-master:
    Container ID: docker://9554b697372a6c89e9dd4aa8d8c33bd74e99bfcc9d694d174136d68fb5b3981c
    Image: elasticsearch:8.12.2
    Image ID: docker-pullable://elasticsearch@sha256:b0dfebb1bf16b89b3cbde7f9197f13b5dd3402595a3dccb0163aa33a042a1541
    Port: <none>
    Host Port: <none>
    State: Waiting
      Reason: CrashLoopBackOff
    Last State: Terminated
      Reason: Error
      Exit Code: 1
      Started: Mon, 26 Feb 2024 10:51:06 +0100
      Finished: Mon, 26 Feb 2024 10:51:06 +0100
    Ready: False
    Restart Count: 4
    Limits:
      cpu: 4
      memory: 4Gi
    Requests:
      cpu: 1
      memory: 4Gi
    Readiness: exec [bash -c /mnt/elastic-internal/scripts/readiness-probe-script.sh] delay=60s timeout=1s period=10s #success=1 #failure=3
    Environment:
      READINESS_PROBE_TIMEOUT: 60
    Mounts: <none>
  elasticsearch:
    Container ID: docker://23812d5791e8dd2f61ed98fac9662876de7ce292d0564e6a2cd080f1b66e2afa
    Image: docker.elastic.co/elasticsearch/elasticsearch:8.12.2
    Image ID: docker-pullable://docker.elastic.co/elasticsearch/elasticsearch@sha256:81b3bbd42d4d5e238e7ee31b7e99acab0bac1fca4eb2fa88415bad5482e52b88
    Ports: 9200/TCP, 9300/TCP
    Host Ports: 0/TCP, 0/TCP
    State: Running
      Started: Mon, 26 Feb 2024 10:49:42 +0100
    Ready: False
    Restart Count: 0
    Limits:
      memory: 2Gi
    Requests:
      memory: 2Gi
    Readiness: exec [bash -c /mnt/elastic-internal/scripts/readiness-probe-script.sh] delay=10s timeout=5s period=5s #success=1 #failure=3
    Environment:
      POD_IP: (v1:status.podIP)
      POD_NAME: elasticsearch-cluster-es-master-0 (v1:metadata.name)
      NODE_NAME: (v1:spec.nodeName)
      NAMESPACE: monitoring (v1:metadata.namespace)
      PROBE_PASSWORD_PATH: /mnt/elastic-internal/pod-mounted-users/elastic-internal-probe
      PROBE_USERNAME: elastic-internal-probe
      READINESS_PROBE_PROTOCOL: https
      HEADLESS_SERVICE_NAME: elasticsearch-cluster-es-master
      NSS_SDB_USE_CACHE: no
    Mounts:
      /mnt/elastic-internal/downward-api from downward-api (ro)
      /mnt/elastic-internal/elasticsearch-config from elastic-internal-elasticsearch-config (ro)
      /mnt/elastic-internal/pod-mounted-users from elastic-internal-probe-user (ro)
      /mnt/elastic-internal/scripts from elastic-internal-scripts (ro)
      /mnt/elastic-internal/unicast-hosts from elastic-internal-unicast-hosts (ro)
      /mnt/elastic-internal/xpack-file-realm from elastic-internal-xpack-file-realm (ro)
      /tmp from tmp-volume (rw)
      /usr/share/elasticsearch/bin from elastic-internal-elasticsearch-bin-local (rw)
      /usr/share/elasticsearch/config from elastic-internal-elasticsearch-config-local (rw)
      /usr/share/elasticsearch/config/http-certs from elastic-internal-http-certificates (ro)
      /usr/share/elasticsearch/config/operator from file-settings (ro)
      /usr/share/elasticsearch/config/transport-certs from elastic-internal-transport-certificates (ro)
      /usr/share/elasticsearch/config/transport-remote-certs/ from elastic-internal-remote-certificate-authorities (ro)
      /usr/share/elasticsearch/data from elasticsearch-data (rw)
      /usr/share/elasticsearch/logs from elasticsearch-logs (rw)
      /usr/share/elasticsearch/plugins from elastic-internal-elasticsearch-plugins-local (rw)
Conditions:
  Type Status
  PodReadyToStartContainers True
  Initialized True
  Ready False
  ContainersReady False
  PodScheduled True
Volumes:
  elasticsearch-data:
    Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName: elasticsearch-data-elasticsearch-cluster-es-master-0
    ReadOnly: false
  downward-api:
    Type: DownwardAPI (a volume populated by information about the pod)
    Items:
      metadata.labels -> labels
  elastic-internal-elasticsearch-bin-local:
    Type: EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit: <unset>
  elastic-internal-elasticsearch-config:
    Type: Secret (a volume populated by a Secret)
    SecretName: elasticsearch-cluster-es-master-es-config
    Optional: false
  elastic-internal-elasticsearch-config-local:
    Type: EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit: <unset>
  elastic-internal-elasticsearch-plugins-local:
    Type: EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit: <unset>
  elastic-internal-http-certificates:
    Type: Secret (a volume populated by a Secret)
    SecretName: elasticsearch-cluster-es-http-certs-internal
    Optional: false
  elastic-internal-probe-user:
    Type: Secret (a volume populated by a Secret)
    SecretName: elasticsearch-cluster-es-internal-users
    Optional: false
  elastic-internal-remote-certificate-authorities:
    Type: Secret (a volume populated by a Secret)
    SecretName: elasticsearch-cluster-es-remote-ca
    Optional: false
  elastic-internal-scripts:
    Type: ConfigMap (a volume populated by a ConfigMap)
    Name: elasticsearch-cluster-es-scripts
    Optional: false
  elastic-internal-transport-certificates:
    Type: Secret (a volume populated by a Secret)
    SecretName: elasticsearch-cluster-es-master-es-transport-certs
    Optional: false
  elastic-internal-unicast-hosts:
    Type: ConfigMap (a volume populated by a ConfigMap)
    Name: elasticsearch-cluster-es-unicast-hosts
    Optional: false
  elastic-internal-xpack-file-realm:
    Type: Secret (a volume populated by a Secret)
    SecretName: elasticsearch-cluster-es-xpack-file-realm
    Optional: false
  elasticsearch-logs:
    Type: EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit: <unset>
  file-settings:
    Type: Secret (a volume populated by a Secret)
    SecretName: elasticsearch-cluster-es-file-settings
    Optional: false
  tmp-volume:
    Type: EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit: <unset>
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Warning FailedScheduling 104s default-scheduler 0/1 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
  Normal Scheduled 103s default-scheduler Successfully assigned monitoring/elasticsearch-cluster-es-master-0 to docker-desktop
  Normal Pulled 103s kubelet Container image "docker.elastic.co/elasticsearch/elasticsearch:8.12.2" already present on machine
  Normal Created 103s kubelet Created container elastic-internal-init-filesystem
  Normal Started 103s kubelet Started container elastic-internal-init-filesystem
  Normal Pulled 100s kubelet Container image "docker.elastic.co/elasticsearch/elasticsearch:8.12.2" already present on machine
  Normal Created 100s kubelet Created container elastic-internal-suspend
  Normal Started 100s kubelet Started container elastic-internal-suspend
  Normal Created 99s kubelet Created container elasticsearch
  Normal Pulled 99s kubelet Container image "docker.elastic.co/elasticsearch/elasticsearch:8.12.2" already present on machine
  Normal Started 99s kubelet Started container elasticsearch
  Warning Unhealthy 88s kubelet Readiness probe failed: {"timestamp": "2024-02-26T09:49:53+00:00", "message": "readiness probe failed", "curl_rc": "7"}
  Warning Unhealthy 83s kubelet Readiness probe failed: {"timestamp": "2024-02-26T09:49:58+00:00", "message": "readiness probe failed", "curl_rc": "35"}
  Normal Created 81s (x3 over 99s) kubelet Created container elasticsearch-master
  Normal Started 81s (x3 over 99s) kubelet Started container elasticsearch-master
  Normal Pulled 81s (x3 over 99s) kubelet Container image "elasticsearch:8.12.2" already present on machine
  Warning Unhealthy 81s kubelet Readiness probe failed: {"timestamp": "2024-02-26T09:50:00+00:00", "message": "readiness probe failed", "curl_rc": "35"}
  Warning BackOff 80s (x4 over 97s) kubelet Back-off restarting failed container elasticsearch-master in pod elasticsearch-cluster-es-master-0_monitoring(b39e97d1-0f8d-4600-828d-7c945aa58d78)

```

Then when I log the master pod :

```auto
kubectl logs -n monitoring elasticsearch-cluster-es-master-0
Defaulted container "elasticsearch-master" out of: elasticsearch-master, elasticsearch, elastic-internal-init-filesystem (init), elastic-internal-suspend (init)
/usr/local/bin/docker-entrypoint.sh: line 84: cannot create temp file for here-document: Read-only file system

```

I tried a lot of things as disable my remote storage mount (as you can see in my manifest), disable xpack security (I don't need for moment) and increase readiness initial delay seconds.

I'm new to ELK and k8s.

Thanks for help.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 25, 2024, 9:56am UTC](https://discuss.elastic.co/t/eck-cant-have-green-elasticsearch-cluster/354106/2 "2024-03-25T09:56:36Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.