ImagePullBackOff error while deploying Elastic via ECK

[alytkowski]

I'm trying to follow the tutorial from this page:

Deploy an Elasticsearch cluster | Elastic Cloud on Kubernetes [2.4] | Elastic

, to deploy Elastic on K8s and I get ImagePullbackOff error, pod events below:

Events:

  Type     Reason            Age                  From               Message

  ----     ------            ----                 ----               -------

  Warning  FailedScheduling  4m7s                 default-scheduler  0/1 nodes are available: 1 pod has unbound immediate PersistentVolumeClaims. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.

  Normal   Scheduled         4m5s                 default-scheduler  Successfully assigned default/ifcm-es-default-0 to minikube

  Warning  Failed            2m4s                 kubelet            Failed to pull image "docker.elastic.co/elasticsearch/elasticsearch:8.4.3": rpc error: code = Unknown desc = context deadline exceeded

  Warning  Failed            2m4s                 kubelet            Error: ErrImagePull

  Normal   BackOff           2m (x3 over 2m3s)    kubelet            Back-off pulling image "docker.elastic.co/elasticsearch/elasticsearch:8.4.3"

  Warning  Failed            2m (x3 over 2m3s)    kubelet            Error: ImagePullBackOff

  Normal   Pulling           105s (x2 over 4m3s)  kubelet            Pulling image "docker.elastic.co/elasticsearch/elasticsearch:8.4.3"

On the other thread someone suggested issuing docker pull command, which I did, successfully, but the ImagePullBackOff error still occurs, Kubernetes doesn't use the already downloaded image, tries to pull it and fails.
Output from docker pull:

docker pull docker.elastic.co/elasticsearch/elasticsearch:8.4.3

8.4.3: Pulling from elasticsearch/elasticsearch

Digest: sha256:739ec9d428869f16e9e02247d5082849ebb4302c87e0abf9f70971cbb40c3bab

Status: Image is up to date for docker.elastic.co/elasticsearch/elasticsearch:8.4.3

docker.elastic.co/elasticsearch/elasticsearch:8.4.3

I'm using a VM with Ubuntu installed and a Bridged Network setup. Prior to starting minikube I've also done docker login command successfully.

[kylemclaren (Kyle M.)]

Hi @alytkowski,

Edit: imagePullPolicy: IfNotPresent will work too, but Never is the nuclear option!

Indeed, it looks like there is an open issue with minikube and one of the recommended workarounds is to manually pull the image, as you did.

First, I would recommend that you confirm the image is loaded into the minikube cluster: minikube ssh docker images | grep elasticsearch

Then, you can use a slightly altered version of the manifest that we have in the docs to force k8s to use the local image instead of trying to pull it, by setting imagePullPolicy: Never

cat <<EOF | kubectl apply -f -
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: quickstart
spec:
  version: 8.4.3
  nodeSets:
  - name: default
    count: 1
    config:
      node.store.allow_mmap: false
    podTemplate:
      spec:
        containers:
        - name: elasticsearch
          imagePullPolicy: Never
EOF

Hope this works for you

[alytkowski]

Thank you Kyle! This works for me, it's exactly what I was looking for. Cheers!