Hello,
I try to deploy an elasticsearch cluster on my docker-desktop kubernetes test cluster. The final goal is to deploy it in a k3s cluster on my raspberry pies.
Here my kubernetes manifest :
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: elasticsearch-cluster
namespace: monitoring
spec:
version: 8.12.2
# pour avancer
# https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-node-configuration.html
nodeSets:
- name: master
count: 1
config:
node.roles: ["master"]
xpack.security.enabled: false
xpack.security.http.ssl.enabled: false
xpack.security.transport.ssl.enabled: false
# volumeClaimTemplates:
# - metadata:
# name: elasticsearch-data
# namespace: monitoring
# spec:
# accessModes:
# - ReadWriteOnce
# resources:
# requests:
# storage: 50Gi
# storageClassName: elasticsearch-master-data
podTemplate:
spec:
containers:
- name: elasticsearch-master
image: elasticsearch:8.12.2
readinessProbe:
initialDelaySeconds: 60
exec:
command:
- bash
- -c
- /mnt/elastic-internal/scripts/readiness-probe-script.sh
env:
- name: READINESS_PROBE_TIMEOUT
value: "60"
# volumeMounts:
# - name: elasticsearch-data
# mountPath: /usr/share/elasticsearch/data
resources:
requests:
memory: 4Gi
cpu: 1
limits:
memory: 4Gi
cpu: 4
- name: data
count: 1
config:
node.roles: ["data", "ingest", "ml", "transform"]
xpack.security.enabled: false
xpack.security.http.ssl.enabled: false
xpack.security.transport.ssl.enabled: false
# volumeClaimTemplates:
# - metadata:
# name: elasticsearch-data
# namespace: monitoring
# spec:
# accessModes:
# - ReadWriteOnce
# resources:
# requests:
# storage: 50Gi
# storageClassName: elasticsearch-client-data
podTemplate:
spec:
containers:
- name: elasticsearch-client
image: elasticsearch:8.12.2
readinessProbe:
initialDelaySeconds: 60
exec:
command:
- bash
- -c
- /mnt/elastic-internal/scripts/readiness-probe-script.sh
env:
- name: READINESS_PROBE_TIMEOUT
value: "60"
# volumeMounts:
# - name: elasticsearch-data
# mountPath: /usr/share/elasticsearch/data
resources:
requests:
memory: 4Gi
cpu: 1
limits:
memory: 4Gi
cpu: 4
When I describe the master pod :
kubectl describe -n monitoring pod elasticsearch-cluster-es-master-0
Name: elasticsearch-cluster-es-master-0
Namespace: monitoring
Priority: 0
Service Account: default
Node: docker-desktop/192.168.65.3
Start Time: Mon, 26 Feb 2024 10:49:38 +0100
Labels: apps.kubernetes.io/pod-index=0
common.k8s.elastic.co/type=elasticsearch
controller-revision-hash=elasticsearch-cluster-es-master-76d967c889
elasticsearch.k8s.elastic.co/cluster-name=elasticsearch-cluster
elasticsearch.k8s.elastic.co/http-scheme=https
elasticsearch.k8s.elastic.co/node-data=false
elasticsearch.k8s.elastic.co/node-data_cold=false
elasticsearch.k8s.elastic.co/node-data_content=false
elasticsearch.k8s.elastic.co/node-data_frozen=false
elasticsearch.k8s.elastic.co/node-data_hot=false
elasticsearch.k8s.elastic.co/node-data_warm=false
elasticsearch.k8s.elastic.co/node-ingest=false
elasticsearch.k8s.elastic.co/node-master=true
elasticsearch.k8s.elastic.co/node-ml=false
elasticsearch.k8s.elastic.co/node-remote_cluster_client=false
elasticsearch.k8s.elastic.co/node-transform=false
elasticsearch.k8s.elastic.co/node-voting_only=false
elasticsearch.k8s.elastic.co/statefulset-name=elasticsearch-cluster-es-master
elasticsearch.k8s.elastic.co/version=8.12.2
statefulset.kubernetes.io/pod-name=elasticsearch-cluster-es-master-0
Annotations: co.elastic.logs/module: elasticsearch
elasticsearch.k8s.elastic.co/config-hash: 1377360743
policy.k8s.elastic.co/elasticsearch-config-mounts-hash:
update.k8s.elastic.co/timestamp: 2024-02-26T09:49:39.681958207Z
Status: Running
IP: 10.1.1.39
IPs:
IP: 10.1.1.39
Controlled By: StatefulSet/elasticsearch-cluster-es-master
Init Containers:
elastic-internal-init-filesystem:
Container ID: docker://302e25d7fe47d93262e5b14171af00fd668e3f89d91f40ec8e29f4f9367b322b
Image: docker.elastic.co/elasticsearch/elasticsearch:8.12.2
Image ID: docker-pullable://docker.elastic.co/elasticsearch/elasticsearch@sha256:81b3bbd42d4d5e238e7ee31b7e99acab0bac1fca4eb2fa88415bad5482e52b88
Port: <none>
Host Port: <none>
Command:
bash
-c
/mnt/elastic-internal/scripts/prepare-fs.sh
State: Terminated
Reason: Completed
Exit Code: 0
Started: Mon, 26 Feb 2024 10:49:38 +0100
Finished: Mon, 26 Feb 2024 10:49:40 +0100
Ready: True
Restart Count: 0
Limits:
cpu: 100m
memory: 50Mi
Requests:
cpu: 100m
memory: 50Mi
Environment:
POD_IP: (v1:status.podIP)
POD_NAME: elasticsearch-cluster-es-master-0 (v1:metadata.name)
NODE_NAME: (v1:spec.nodeName)
NAMESPACE: monitoring (v1:metadata.namespace)
PROBE_PASSWORD_PATH: /mnt/elastic-internal/pod-mounted-users/elastic-internal-probe
PROBE_USERNAME: elastic-internal-probe
READINESS_PROBE_PROTOCOL: https
HEADLESS_SERVICE_NAME: elasticsearch-cluster-es-master
NSS_SDB_USE_CACHE: no
Mounts:
/mnt/elastic-internal/downward-api from downward-api (ro)
/mnt/elastic-internal/elasticsearch-bin-local from elastic-internal-elasticsearch-bin-local (rw)
/mnt/elastic-internal/elasticsearch-config from elastic-internal-elasticsearch-config (ro)
/mnt/elastic-internal/elasticsearch-config-local from elastic-internal-elasticsearch-config-local (rw)
/mnt/elastic-internal/elasticsearch-plugins-local from elastic-internal-elasticsearch-plugins-local (rw)
/mnt/elastic-internal/pod-mounted-users from elastic-internal-probe-user (ro)
/mnt/elastic-internal/scripts from elastic-internal-scripts (ro)
/mnt/elastic-internal/transport-certificates from elastic-internal-transport-certificates (ro)
/mnt/elastic-internal/unicast-hosts from elastic-internal-unicast-hosts (ro)
/mnt/elastic-internal/xpack-file-realm from elastic-internal-xpack-file-realm (ro)
/tmp from tmp-volume (rw)
/usr/share/elasticsearch/config/http-certs from elastic-internal-http-certificates (ro)
/usr/share/elasticsearch/config/operator from file-settings (ro)
/usr/share/elasticsearch/config/transport-remote-certs/ from elastic-internal-remote-certificate-authorities (ro)
/usr/share/elasticsearch/data from elasticsearch-data (rw)
/usr/share/elasticsearch/logs from elasticsearch-logs (rw)
elastic-internal-suspend:
Container ID: docker://5a88f4aa4dd11540da7ecdcf9b07bf78e4233750ca09ef0ba63eed2d9d7d099c
Image: docker.elastic.co/elasticsearch/elasticsearch:8.12.2
Image ID: docker-pullable://docker.elastic.co/elasticsearch/elasticsearch@sha256:81b3bbd42d4d5e238e7ee31b7e99acab0bac1fca4eb2fa88415bad5482e52b88
Port: <none>
Host Port: <none>
Command:
bash
-c
/mnt/elastic-internal/scripts/suspend.sh
State: Terminated
Reason: Completed
Exit Code: 0
Started: Mon, 26 Feb 2024 10:49:41 +0100
Finished: Mon, 26 Feb 2024 10:49:41 +0100
Ready: True
Restart Count: 0
Limits:
memory: 2Gi
Requests:
memory: 2Gi
Environment:
POD_IP: (v1:status.podIP)
POD_NAME: elasticsearch-cluster-es-master-0 (v1:metadata.name)
NODE_NAME: (v1:spec.nodeName)
NAMESPACE: monitoring (v1:metadata.namespace)
PROBE_PASSWORD_PATH: /mnt/elastic-internal/pod-mounted-users/elastic-internal-probe
PROBE_USERNAME: elastic-internal-probe
READINESS_PROBE_PROTOCOL: https
HEADLESS_SERVICE_NAME: elasticsearch-cluster-es-master
NSS_SDB_USE_CACHE: no
Mounts:
/mnt/elastic-internal/downward-api from downward-api (ro)
/mnt/elastic-internal/elasticsearch-config from elastic-internal-elasticsearch-config (ro)
/mnt/elastic-internal/pod-mounted-users from elastic-internal-probe-user (ro)
/mnt/elastic-internal/scripts from elastic-internal-scripts (ro)
/mnt/elastic-internal/unicast-hosts from elastic-internal-unicast-hosts (ro)
/mnt/elastic-internal/xpack-file-realm from elastic-internal-xpack-file-realm (ro)
/tmp from tmp-volume (rw)
/usr/share/elasticsearch/bin from elastic-internal-elasticsearch-bin-local (rw)
/usr/share/elasticsearch/config from elastic-internal-elasticsearch-config-local (rw)
/usr/share/elasticsearch/config/http-certs from elastic-internal-http-certificates (ro)
/usr/share/elasticsearch/config/operator from file-settings (ro)
/usr/share/elasticsearch/config/transport-certs from elastic-internal-transport-certificates (ro)
/usr/share/elasticsearch/config/transport-remote-certs/ from elastic-internal-remote-certificate-authorities (ro)
/usr/share/elasticsearch/data from elasticsearch-data (rw)
/usr/share/elasticsearch/logs from elasticsearch-logs (rw)
/usr/share/elasticsearch/plugins from elastic-internal-elasticsearch-plugins-local (rw)
Containers:
elasticsearch-master:
Container ID: docker://9554b697372a6c89e9dd4aa8d8c33bd74e99bfcc9d694d174136d68fb5b3981c
Image: elasticsearch:8.12.2
Image ID: docker-pullable://elasticsearch@sha256:b0dfebb1bf16b89b3cbde7f9197f13b5dd3402595a3dccb0163aa33a042a1541
Port: <none>
Host Port: <none>
State: Waiting
Reason: CrashLoopBackOff
Last State: Terminated
Reason: Error
Exit Code: 1
Started: Mon, 26 Feb 2024 10:51:06 +0100
Finished: Mon, 26 Feb 2024 10:51:06 +0100
Ready: False
Restart Count: 4
Limits:
cpu: 4
memory: 4Gi
Requests:
cpu: 1
memory: 4Gi
Readiness: exec [bash -c /mnt/elastic-internal/scripts/readiness-probe-script.sh] delay=60s timeout=1s period=10s #success=1 #failure=3
Environment:
READINESS_PROBE_TIMEOUT: 60
Mounts: <none>
elasticsearch:
Container ID: docker://23812d5791e8dd2f61ed98fac9662876de7ce292d0564e6a2cd080f1b66e2afa
Image: docker.elastic.co/elasticsearch/elasticsearch:8.12.2
Image ID: docker-pullable://docker.elastic.co/elasticsearch/elasticsearch@sha256:81b3bbd42d4d5e238e7ee31b7e99acab0bac1fca4eb2fa88415bad5482e52b88
Ports: 9200/TCP, 9300/TCP
Host Ports: 0/TCP, 0/TCP
State: Running
Started: Mon, 26 Feb 2024 10:49:42 +0100
Ready: False
Restart Count: 0
Limits:
memory: 2Gi
Requests:
memory: 2Gi
Readiness: exec [bash -c /mnt/elastic-internal/scripts/readiness-probe-script.sh] delay=10s timeout=5s period=5s #success=1 #failure=3
Environment:
POD_IP: (v1:status.podIP)
POD_NAME: elasticsearch-cluster-es-master-0 (v1:metadata.name)
NODE_NAME: (v1:spec.nodeName)
NAMESPACE: monitoring (v1:metadata.namespace)
PROBE_PASSWORD_PATH: /mnt/elastic-internal/pod-mounted-users/elastic-internal-probe
PROBE_USERNAME: elastic-internal-probe
READINESS_PROBE_PROTOCOL: https
HEADLESS_SERVICE_NAME: elasticsearch-cluster-es-master
NSS_SDB_USE_CACHE: no
Mounts:
/mnt/elastic-internal/downward-api from downward-api (ro)
/mnt/elastic-internal/elasticsearch-config from elastic-internal-elasticsearch-config (ro)
/mnt/elastic-internal/pod-mounted-users from elastic-internal-probe-user (ro)
/mnt/elastic-internal/scripts from elastic-internal-scripts (ro)
/mnt/elastic-internal/unicast-hosts from elastic-internal-unicast-hosts (ro)
/mnt/elastic-internal/xpack-file-realm from elastic-internal-xpack-file-realm (ro)
/tmp from tmp-volume (rw)
/usr/share/elasticsearch/bin from elastic-internal-elasticsearch-bin-local (rw)
/usr/share/elasticsearch/config from elastic-internal-elasticsearch-config-local (rw)
/usr/share/elasticsearch/config/http-certs from elastic-internal-http-certificates (ro)
/usr/share/elasticsearch/config/operator from file-settings (ro)
/usr/share/elasticsearch/config/transport-certs from elastic-internal-transport-certificates (ro)
/usr/share/elasticsearch/config/transport-remote-certs/ from elastic-internal-remote-certificate-authorities (ro)
/usr/share/elasticsearch/data from elasticsearch-data (rw)
/usr/share/elasticsearch/logs from elasticsearch-logs (rw)
/usr/share/elasticsearch/plugins from elastic-internal-elasticsearch-plugins-local (rw)
Conditions:
Type Status
PodReadyToStartContainers True
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
elasticsearch-data:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: elasticsearch-data-elasticsearch-cluster-es-master-0
ReadOnly: false
downward-api:
Type: DownwardAPI (a volume populated by information about the pod)
Items:
metadata.labels -> labels
elastic-internal-elasticsearch-bin-local:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
elastic-internal-elasticsearch-config:
Type: Secret (a volume populated by a Secret)
SecretName: elasticsearch-cluster-es-master-es-config
Optional: false
elastic-internal-elasticsearch-config-local:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
elastic-internal-elasticsearch-plugins-local:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
elastic-internal-http-certificates:
Type: Secret (a volume populated by a Secret)
SecretName: elasticsearch-cluster-es-http-certs-internal
Optional: false
elastic-internal-probe-user:
Type: Secret (a volume populated by a Secret)
SecretName: elasticsearch-cluster-es-internal-users
Optional: false
elastic-internal-remote-certificate-authorities:
Type: Secret (a volume populated by a Secret)
SecretName: elasticsearch-cluster-es-remote-ca
Optional: false
elastic-internal-scripts:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: elasticsearch-cluster-es-scripts
Optional: false
elastic-internal-transport-certificates:
Type: Secret (a volume populated by a Secret)
SecretName: elasticsearch-cluster-es-master-es-transport-certs
Optional: false
elastic-internal-unicast-hosts:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: elasticsearch-cluster-es-unicast-hosts
Optional: false
elastic-internal-xpack-file-realm:
Type: Secret (a volume populated by a Secret)
SecretName: elasticsearch-cluster-es-xpack-file-realm
Optional: false
elasticsearch-logs:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
file-settings:
Type: Secret (a volume populated by a Secret)
SecretName: elasticsearch-cluster-es-file-settings
Optional: false
tmp-volume:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedScheduling 104s default-scheduler 0/1 nodes are available: pod has unbound immediate PersistentVolumeClaims. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
Normal Scheduled 103s default-scheduler Successfully assigned monitoring/elasticsearch-cluster-es-master-0 to docker-desktop
Normal Pulled 103s kubelet Container image "docker.elastic.co/elasticsearch/elasticsearch:8.12.2" already present on machine
Normal Created 103s kubelet Created container elastic-internal-init-filesystem
Normal Started 103s kubelet Started container elastic-internal-init-filesystem
Normal Pulled 100s kubelet Container image "docker.elastic.co/elasticsearch/elasticsearch:8.12.2" already present on machine
Normal Created 100s kubelet Created container elastic-internal-suspend
Normal Started 100s kubelet Started container elastic-internal-suspend
Normal Created 99s kubelet Created container elasticsearch
Normal Pulled 99s kubelet Container image "docker.elastic.co/elasticsearch/elasticsearch:8.12.2" already present on machine
Normal Started 99s kubelet Started container elasticsearch
Warning Unhealthy 88s kubelet Readiness probe failed: {"timestamp": "2024-02-26T09:49:53+00:00", "message": "readiness probe failed", "curl_rc": "7"}
Warning Unhealthy 83s kubelet Readiness probe failed: {"timestamp": "2024-02-26T09:49:58+00:00", "message": "readiness probe failed", "curl_rc": "35"}
Normal Created 81s (x3 over 99s) kubelet Created container elasticsearch-master
Normal Started 81s (x3 over 99s) kubelet Started container elasticsearch-master
Normal Pulled 81s (x3 over 99s) kubelet Container image "elasticsearch:8.12.2" already present on machine
Warning Unhealthy 81s kubelet Readiness probe failed: {"timestamp": "2024-02-26T09:50:00+00:00", "message": "readiness probe failed", "curl_rc": "35"}
Warning BackOff 80s (x4 over 97s) kubelet Back-off restarting failed container elasticsearch-master in pod elasticsearch-cluster-es-master-0_monitoring(b39e97d1-0f8d-4600-828d-7c945aa58d78)
Then when I log the master pod :
kubectl logs -n monitoring elasticsearch-cluster-es-master-0
Defaulted container "elasticsearch-master" out of: elasticsearch-master, elasticsearch, elastic-internal-init-filesystem (init), elastic-internal-suspend (init)
/usr/local/bin/docker-entrypoint.sh: line 84: cannot create temp file for here-document: Read-only file system
I tried a lot of things as disable my remote storage mount (as you can see in my manifest), disable xpack security (I don't need for moment) and increase readiness initial delay seconds.
I'm new to ELK and k8s.
Thanks for help.