Error: ImagePullBackOff

Hello,

I want to deploy quickstart example of elk, kibana, fleet server using this example but pods cannot start because

NAMESPACE        NAME                                        READY   STATUS                  RESTARTS        AGE
default          elasticsearch-quickstart-es-default-0       0/1     Init:ImagePullBackOff   0               8m
default          kibana-quickstart-kb-559ff6d669-cdmfh       1/1     Running                 0               7m54s
default          quickstart-agent-bzgfn                      0/1     ImagePullBackOff        0               43m
elastic-system   elastic-operator-0                          1/1     Running                 6 (9m33s ago)   5h2m
ingress-nginx    ingress-nginx-admission-create-x6zdh        0/1     Completed               0               4h59m
ingress-nginx    ingress-nginx-admission-patch-r4n8m         0/1     Completed               1               4h59m
ingress-nginx    ingress-nginx-controller-5959f988fd-gs2dc   1/1     Running                 4 (9m32s ago)   4h59m
kube-system      coredns-565d847f94-jrhpr                    1/1     Running                 4 (9m33s ago)   5h3m
kube-system      etcd-minikube                               1/1     Running                 4 (9m33s ago)   5h3m
kube-system      kube-apiserver-minikube                     1/1     Running                 4 (9m33s ago)   5h3m
kube-system      kube-controller-manager-minikube            1/1     Running                 4 (9m33s ago)   5h3m
kube-system      kube-proxy-tzpqb                            1/1     Running                 4 (9m32s ago)   5h3m
kube-system      kube-scheduler-minikube                     1/1     Running                 4 (9m33s ago)   5h3m
kube-system      storage-provisioner                         1/1     Running                 7 (9m32s ago)   5h3m
Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  6m10s                  default-scheduler  Successfully assigned default/elasticsearch-quickstart-es-default-0 to minikube
  Warning  Failed     2m33s                  kubelet            Failed to pull image "docker.elastic.co/elasticsearch/elasticsearch:8.4.1": rpc error: code = Unknown desc = context deadline exceeded
  Warning  Failed     2m33s                  kubelet            Error: ErrImagePull
  Normal   BackOff    2m31s (x3 over 2m33s)  kubelet            Back-off pulling image "docker.elastic.co/elasticsearch/elasticsearch:8.4.1"
  Warning  Failed     2m31s (x3 over 2m33s)  kubelet            Error: ImagePullBackOff
  Normal   Pulling    2m17s (x2 over 6m4s)   kubelet            Pulling image "docker.elastic.co/elasticsearch/elasticsearch:8.4.1"

I was testing version 8.4.0, 8.4.1, 8.4.2, 8.4.3 but all cannot be pulled. Which version should I use for it to works?

Hey @Adriann, this is a known issue with Minikube. Check out this topic which was recently solved: ImagePullBackOff error while deploying Elastic via ECK - #2 by kylemclaren

1 Like

Hello Kyle,

thank you for your answer.

To be honest I don't understand what should I change based on that topic.

In short, you need to manually pull the images down to the minkiube cluster: minikube ssh docker pull docker.elastic.co/../..

Then, force k8s to use the local images and not try to pull them using imagePullPolicy: Never. So in the linked topic, the example is:

cat <<EOF | kubectl apply -f -
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: quickstart
spec:
  version: 8.4.3
  nodeSets:
  - name: default
    count: 1
    config:
      node.store.allow_mmap: false
    podTemplate:
      spec:
        containers:
        - name: elasticsearch
          imagePullPolicy: Never
EOF

You'll just need to apply that imagePullPolicy to the Fleet quickstart.

I have downloaded the imgs


minikube ssh docker pull docker.elastic.co/elasticsearch/elasticsearch:8.4.3
minikube ssh docker pull docker.elastic.co/beats/elastic-agent:8.4.3

I did the change

apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: fleet-server-quickstart
  namespace: default
spec:
  version: 8.4.3
  kibanaRef:
    name: kibana-quickstart
  elasticsearchRefs:
  - name: elasticsearch-quickstart
  mode: fleet
  fleetServerEnabled: true
  deployment:
    replicas: 1
    podTemplate:
      spec:
        serviceAccountName: elastic-agent
        automountServiceAccountToken: true
        securityContext:
          runAsUser: 0
        containers:
        - name: agent
          imagePullPolicy: Never
---
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
  name: elastic-agent-quickstart
  namespace: default
spec:
  version: 8.4.3
  kibanaRef:
    name: kibana-quickstart
  fleetServerRef:
    name: fleet-server-quickstart
  mode: fleet
  daemonSet:
    podTemplate:
      spec:
        serviceAccountName: elastic-agent
        automountServiceAccountToken: true
        securityContext:
          runAsUser: 0
      containers:
      - name: agent
        imagePullPolicy: Never
---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana-quickstart
  namespace: default
spec:
  version: 8.4.3
  count: 1
  elasticsearchRef:
    name: elasticsearch-quickstart
  config:
    xpack.fleet.agents.elasticsearch.hosts: ["https://elasticsearch-quickstart-es-http.default.svc:9200"]
    xpack.fleet.agents.fleet_server.hosts: ["https://fleet-server-quickstart-agent-http.default.svc:8220"]
    xpack.fleet.packages:
      - name: system
        version: latest
      - name: elastic_agent
        version: latest
      - name: fleet_server
        version: latest
    xpack.fleet.agentPolicies:
      - name: Fleet Server on ECK policy
        id: eck-fleet-server
        is_default_fleet_server: true
        namespace: default
        monitoring_enabled:
          - logs
          - metrics
        unenroll_timeout: 900
        package_policies:
        - name: fleet_server-1
          id: fleet_server-1
          package:
            name: fleet_server
      - name: Elastic Agent on ECK policy
        id: eck-agent
        namespace: default
        monitoring_enabled:
          - logs
          - metrics
        unenroll_timeout: 900
        is_default: true
        package_policies:
          - name: system-1
            id: system-1
            package:
              name: system
  http:
    service:
      spec:
        ports:
          - port: 5601
            targetPort: 5601
        type: LoadBalancer
---
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: elasticsearch-quickstart
  namespace: default
spec:
  version: 8.4.3
  nodeSets:
  - name: default
    count: 1
    config:
      node.store.allow_mmap: false
    podTemplate:
      spec:
        containers:
        - name: elasticsearch
          imagePullPolicy: Never
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: elastic-agent
rules:
- apiGroups: [""] # "" indicates the core API group
  resources:
  - pods
  - nodes
  - namespaces
  verbs:
  - get
  - watch
  - list
- apiGroups: ["coordination.k8s.io"]
  resources:
  - leases
  verbs:
  - get
  - create
  - update
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: elastic-agent
  namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: elastic-agent
subjects:
- kind: ServiceAccount
  name: elastic-agent
  namespace: default
roleRef:
  kind: ClusterRole
  name: elastic-agent
  apiGroup: rbac.authorization.k8s.io

This is the output

66s         Warning   AssociationError          agent/fleet-server-quickstart                                                    Association backend for kibana is not configured
2m10s       Warning   AssociationError          agent/fleet-server-quickstart                                                    Association backend for elasticsearch is not configured
2m10s       Normal    AssociationStatusChange   agent/fleet-server-quickstart                                                    Association status changed from [] to [Pending]
2m9s        Normal    AssociationStatusChange   agent/fleet-server-quickstart                                                    Association status changed from [] to [default/elasticsearch-quickstart: Established]
66s         Normal    AssociationStatusChange   agent/elastic-agent-quickstart                                                   Association status changed from [] to [Pending]
66s         Normal    AssociationStatusChange   agent/elastic-agent-quickstart                                                   Association status changed from [] to [Pending]
66s         Warning   AssociationError          agent/elastic-agent-quickstart                                                   Association backend for fleetserver is not configured
66s         Warning   AssociationError          agent/elastic-agent-quickstart                                                   Association backend for kibana is not configured

k get po -A --watch

NAMESPACE        NAME                                        READY   STATUS      RESTARTS        AGE
default          elasticsearch-quickstart-es-default-0       1/1     Running     1 (2m20s ago)   3h11m
default          kibana-quickstart-kb-559ff6d669-cdmfh       0/1     Running     1 (2m20s ago)   3h11m
default          quickstart-agent-5mj6z                      0/1     Error       1 (17s ago)     22s
elastic-system   elastic-operator-0                          1/1     Running     8 (31s ago)     8h
ingress-nginx    ingress-nginx-admission-create-x6zdh        0/1     Completed   0               8h
ingress-nginx    ingress-nginx-admission-patch-r4n8m         0/1     Completed   1               8h
ingress-nginx    ingress-nginx-controller-5959f988fd-gs2dc   1/1     Running     5 (2m20s ago)   8h
kube-system      coredns-565d847f94-jrhpr                    1/1     Running     5 (2m20s ago)   8h
kube-system      etcd-minikube                               1/1     Running     5 (2m20s ago)   8h
kube-system      kube-apiserver-minikube                     1/1     Running     5 (2m20s ago)   8h
kube-system      kube-controller-manager-minikube            1/1     Running     5 (2m20s ago)   8h
kube-system      kube-proxy-tzpqb                            1/1     Running     5 (2m20s ago)   8h
kube-system      kube-scheduler-minikube                     1/1     Running     5 (2m20s ago)   8h
kube-system      storage-provisioner                         1/1     Running     10 (31s ago)    8h
default          quickstart-agent-5mj6z                      0/1     CrashLoopBackOff   1 (15s ago)     23s
default          kibana-quickstart-kb-559ff6d669-cdmfh       0/1     Terminating        1 (2m22s ago)   3h11m
default          elasticsearch-quickstart-es-default-0       1/1     Running            1 (2m23s ago)   3h11m
default          elasticsearch-quickstart-es-default-0       1/1     Running            1 (2m23s ago)   3h11m
default          quickstart-agent-5mj6z                      0/1     Error              2 (17s ago)     25s
default          elasticsearch-quickstart-es-default-0       1/1     Terminating        1 (2m24s ago)   3h11m
default          kibana-quickstart-kb-559ff6d669-cdmfh       0/1     Terminating        1 (2m24s ago)   3h11m
default          kibana-quickstart-kb-559ff6d669-cdmfh       0/1     Terminating        1 (2m24s ago)   3h11m
default          kibana-quickstart-kb-559ff6d669-cdmfh       0/1     Terminating        1 (2m24s ago)   3h11m
default          kibana-quickstart-kb-54597746b7-57v89       0/1     Pending            0               0s
default          kibana-quickstart-kb-54597746b7-57v89       0/1     Pending            0               0s
default          kibana-quickstart-kb-54597746b7-57v89       0/1     Init:0/1           0               1s
default          kibana-quickstart-kb-54597746b7-57v89       0/1     PodInitializing    0               3s
default          kibana-quickstart-kb-54597746b7-57v89       0/1     Running            0               4s
default          quickstart-agent-5mj6z                      0/1     CrashLoopBackOff   2 (12s ago)     37s
default          quickstart-agent-5mj6z                      0/1     Error              3 (30s ago)     55s

k logs quickstart-agent-5mj6z

Error: preparing STATE_PATH(/usr/share/elastic-agent/state) failed: mkdir /usr/share/elastic-agent/state/data: permission denied
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/7.17/fleet-troubleshooting.html

Agnet's pods still do not start but not the pod state is an error.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.