We are using Helm chart from GitHub - elastic/helm-charts: You know, for Kubernetes to deploy Elasticsearch & Kibana in AWS EKS cluster (Elasticsearch & kibana version: 7.15.0). While upgrading Elasticsearch version to 7.16.1, we found that the Elasticsearch pods are failing health checks.
When we described one of the Elasticsearch-master pods, it provided following details:
Events:
Type Reason Age From Message
Normal Scheduled 10m default-scheduler Successfully assigned Elasticsearch-master-0 to ip-10-0-0-27.us-east-2.compute.internal
Normal SuccessfulAttachVolume 10m attachdetach-controller AttachVolume.Attach succeeded for volume "pvc-8b96e33f-e06b-41f5-a66e-30d44c2c5787"
Normal Pulling 10m kubelet Pulling image "docker.elastic.co/elasticsearch/elasticsearch:7.16.1"
Normal Pulled 10m kubelet Successfully pulled image "docker.elastic.co/elasticsearch/elasticsearch:7.16.1" in 613.341063ms
Normal Created 10m kubelet Created container configure-sysctl
Normal Started 10m kubelet Started container configure-sysctl
Normal Pulling 10m kubelet Pulling image "docker.elastic.co/elasticsearch/elasticsearch:7.16.1"
Normal Pulled 10m kubelet Successfully pulled image "docker.elastic.co/elasticsearch/elasticsearch:7.16.1" in 561.219305ms
Normal Created 10m kubelet Created container Elasticsearch
Normal Started 10m kubelet Started container Elasticsearch
Warning Unhealthy 8m49s (x8 over 9m59s) kubelet Readiness probe failed: Waiting for Elasticsearch cluster to become ready (request params: "timeout=1s" )
Cluster is not yet ready (request params: "timeout=1s" )
Warning Unhealthy 9s (x51 over 8m29s) kubelet Readiness probe failed: Elasticsearch is already running, lets check the node is healthy
curl --output /dev/null -k -XGET -s -w '%{http_code}' ${BASIC_AUTH} http://127.0.0.1:9200/ failed with HTTP code 200
sh: 30: [[: not found
sh: 35: [[: not found
sh: 37: [[: not found
Has anyone faced this issue before ? If so, could you please provide suggestions ?