Adriann
October 28, 2022, 1:37pm
1
Hello,
I want to deploy quickstart example of elk, kibana, fleet server using this example but pods cannot start because
NAMESPACE NAME READY STATUS RESTARTS AGE
default elasticsearch-quickstart-es-default-0 0/1 Init:ImagePullBackOff 0 8m
default kibana-quickstart-kb-559ff6d669-cdmfh 1/1 Running 0 7m54s
default quickstart-agent-bzgfn 0/1 ImagePullBackOff 0 43m
elastic-system elastic-operator-0 1/1 Running 6 (9m33s ago) 5h2m
ingress-nginx ingress-nginx-admission-create-x6zdh 0/1 Completed 0 4h59m
ingress-nginx ingress-nginx-admission-patch-r4n8m 0/1 Completed 1 4h59m
ingress-nginx ingress-nginx-controller-5959f988fd-gs2dc 1/1 Running 4 (9m32s ago) 4h59m
kube-system coredns-565d847f94-jrhpr 1/1 Running 4 (9m33s ago) 5h3m
kube-system etcd-minikube 1/1 Running 4 (9m33s ago) 5h3m
kube-system kube-apiserver-minikube 1/1 Running 4 (9m33s ago) 5h3m
kube-system kube-controller-manager-minikube 1/1 Running 4 (9m33s ago) 5h3m
kube-system kube-proxy-tzpqb 1/1 Running 4 (9m32s ago) 5h3m
kube-system kube-scheduler-minikube 1/1 Running 4 (9m33s ago) 5h3m
kube-system storage-provisioner 1/1 Running 7 (9m32s ago) 5h3m
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 6m10s default-scheduler Successfully assigned default/elasticsearch-quickstart-es-default-0 to minikube
Warning Failed 2m33s kubelet Failed to pull image "docker.elastic.co/elasticsearch/elasticsearch:8.4.1": rpc error: code = Unknown desc = context deadline exceeded
Warning Failed 2m33s kubelet Error: ErrImagePull
Normal BackOff 2m31s (x3 over 2m33s) kubelet Back-off pulling image "docker.elastic.co/elasticsearch/elasticsearch:8.4.1"
Warning Failed 2m31s (x3 over 2m33s) kubelet Error: ImagePullBackOff
Normal Pulling 2m17s (x2 over 6m4s) kubelet Pulling image "docker.elastic.co/elasticsearch/elasticsearch:8.4.1"
I was testing version 8.4.0, 8.4.1, 8.4.2, 8.4.3 but all cannot be pulled. Which version should I use for it to works?
Hey @Adriann , this is a known issue with Minikube. Check out this topic which was recently solved: ImagePullBackOff error while deploying Elastic via ECK - #2 by kylemclaren
Adriann
October 28, 2022, 2:10pm
3
Hello Kyle,
thank you for your answer.
To be honest I don't understand what should I change based on that topic.
In short, you need to manually pull the images down to the minkiube cluster: minikube ssh docker pull docker.elastic.co/../..
Then, force k8s to use the local images and not try to pull them using imagePullPolicy: Never. So in the linked topic, the example is:
cat <<EOF | kubectl apply -f -
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: quickstart
spec:
version: 8.4.3
nodeSets:
- name: default
count: 1
config:
node.store.allow_mmap: false
podTemplate:
spec:
containers:
- name: elasticsearch
imagePullPolicy: Never
EOF
You'll just need to apply that imagePullPolicy to the Fleet quickstart.
Adriann
October 28, 2022, 4:45pm
6
I have downloaded the imgs
minikube ssh docker pull docker.elastic.co/elasticsearch/elasticsearch:8.4.3
minikube ssh docker pull docker.elastic.co/beats/elastic-agent:8.4.3
I did the change
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
name: fleet-server-quickstart
namespace: default
spec:
version: 8.4.3
kibanaRef:
name: kibana-quickstart
elasticsearchRefs:
- name: elasticsearch-quickstart
mode: fleet
fleetServerEnabled: true
deployment:
replicas: 1
podTemplate:
spec:
serviceAccountName: elastic-agent
automountServiceAccountToken: true
securityContext:
runAsUser: 0
containers:
- name: agent
imagePullPolicy: Never
---
apiVersion: agent.k8s.elastic.co/v1alpha1
kind: Agent
metadata:
name: elastic-agent-quickstart
namespace: default
spec:
version: 8.4.3
kibanaRef:
name: kibana-quickstart
fleetServerRef:
name: fleet-server-quickstart
mode: fleet
daemonSet:
podTemplate:
spec:
serviceAccountName: elastic-agent
automountServiceAccountToken: true
securityContext:
runAsUser: 0
containers:
- name: agent
imagePullPolicy: Never
---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
name: kibana-quickstart
namespace: default
spec:
version: 8.4.3
count: 1
elasticsearchRef:
name: elasticsearch-quickstart
config:
xpack.fleet.agents.elasticsearch.hosts: ["https://elasticsearch-quickstart-es-http.default.svc:9200"]
xpack.fleet.agents.fleet_server.hosts: ["https://fleet-server-quickstart-agent-http.default.svc:8220"]
xpack.fleet.packages:
- name: system
version: latest
- name: elastic_agent
version: latest
- name: fleet_server
version: latest
xpack.fleet.agentPolicies:
- name: Fleet Server on ECK policy
id: eck-fleet-server
is_default_fleet_server: true
namespace: default
monitoring_enabled:
- logs
- metrics
unenroll_timeout: 900
package_policies:
- name: fleet_server-1
id: fleet_server-1
package:
name: fleet_server
- name: Elastic Agent on ECK policy
id: eck-agent
namespace: default
monitoring_enabled:
- logs
- metrics
unenroll_timeout: 900
is_default: true
package_policies:
- name: system-1
id: system-1
package:
name: system
http:
service:
spec:
ports:
- port: 5601
targetPort: 5601
type: LoadBalancer
---
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: elasticsearch-quickstart
namespace: default
spec:
version: 8.4.3
nodeSets:
- name: default
count: 1
config:
node.store.allow_mmap: false
podTemplate:
spec:
containers:
- name: elasticsearch
imagePullPolicy: Never
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: elastic-agent
rules:
- apiGroups: [""] # "" indicates the core API group
resources:
- pods
- nodes
- namespaces
verbs:
- get
- watch
- list
- apiGroups: ["coordination.k8s.io"]
resources:
- leases
verbs:
- get
- create
- update
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: elastic-agent
namespace: default
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: elastic-agent
subjects:
- kind: ServiceAccount
name: elastic-agent
namespace: default
roleRef:
kind: ClusterRole
name: elastic-agent
apiGroup: rbac.authorization.k8s.io
This is the output
66s Warning AssociationError agent/fleet-server-quickstart Association backend for kibana is not configured
2m10s Warning AssociationError agent/fleet-server-quickstart Association backend for elasticsearch is not configured
2m10s Normal AssociationStatusChange agent/fleet-server-quickstart Association status changed from [] to [Pending]
2m9s Normal AssociationStatusChange agent/fleet-server-quickstart Association status changed from [] to [default/elasticsearch-quickstart: Established]
66s Normal AssociationStatusChange agent/elastic-agent-quickstart Association status changed from [] to [Pending]
66s Normal AssociationStatusChange agent/elastic-agent-quickstart Association status changed from [] to [Pending]
66s Warning AssociationError agent/elastic-agent-quickstart Association backend for fleetserver is not configured
66s Warning AssociationError agent/elastic-agent-quickstart Association backend for kibana is not configured
k get po -A --watch
NAMESPACE NAME READY STATUS RESTARTS AGE
default elasticsearch-quickstart-es-default-0 1/1 Running 1 (2m20s ago) 3h11m
default kibana-quickstart-kb-559ff6d669-cdmfh 0/1 Running 1 (2m20s ago) 3h11m
default quickstart-agent-5mj6z 0/1 Error 1 (17s ago) 22s
elastic-system elastic-operator-0 1/1 Running 8 (31s ago) 8h
ingress-nginx ingress-nginx-admission-create-x6zdh 0/1 Completed 0 8h
ingress-nginx ingress-nginx-admission-patch-r4n8m 0/1 Completed 1 8h
ingress-nginx ingress-nginx-controller-5959f988fd-gs2dc 1/1 Running 5 (2m20s ago) 8h
kube-system coredns-565d847f94-jrhpr 1/1 Running 5 (2m20s ago) 8h
kube-system etcd-minikube 1/1 Running 5 (2m20s ago) 8h
kube-system kube-apiserver-minikube 1/1 Running 5 (2m20s ago) 8h
kube-system kube-controller-manager-minikube 1/1 Running 5 (2m20s ago) 8h
kube-system kube-proxy-tzpqb 1/1 Running 5 (2m20s ago) 8h
kube-system kube-scheduler-minikube 1/1 Running 5 (2m20s ago) 8h
kube-system storage-provisioner 1/1 Running 10 (31s ago) 8h
default quickstart-agent-5mj6z 0/1 CrashLoopBackOff 1 (15s ago) 23s
default kibana-quickstart-kb-559ff6d669-cdmfh 0/1 Terminating 1 (2m22s ago) 3h11m
default elasticsearch-quickstart-es-default-0 1/1 Running 1 (2m23s ago) 3h11m
default elasticsearch-quickstart-es-default-0 1/1 Running 1 (2m23s ago) 3h11m
default quickstart-agent-5mj6z 0/1 Error 2 (17s ago) 25s
default elasticsearch-quickstart-es-default-0 1/1 Terminating 1 (2m24s ago) 3h11m
default kibana-quickstart-kb-559ff6d669-cdmfh 0/1 Terminating 1 (2m24s ago) 3h11m
default kibana-quickstart-kb-559ff6d669-cdmfh 0/1 Terminating 1 (2m24s ago) 3h11m
default kibana-quickstart-kb-559ff6d669-cdmfh 0/1 Terminating 1 (2m24s ago) 3h11m
default kibana-quickstart-kb-54597746b7-57v89 0/1 Pending 0 0s
default kibana-quickstart-kb-54597746b7-57v89 0/1 Pending 0 0s
default kibana-quickstart-kb-54597746b7-57v89 0/1 Init:0/1 0 1s
default kibana-quickstart-kb-54597746b7-57v89 0/1 PodInitializing 0 3s
default kibana-quickstart-kb-54597746b7-57v89 0/1 Running 0 4s
default quickstart-agent-5mj6z 0/1 CrashLoopBackOff 2 (12s ago) 37s
default quickstart-agent-5mj6z 0/1 Error 3 (30s ago) 55s
k logs quickstart-agent-5mj6z
Error: preparing STATE_PATH(/usr/share/elastic-agent/state) failed: mkdir /usr/share/elastic-agent/state/data: permission denied
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/7.17/fleet-troubleshooting.html
Agnet's pods still do not start but not the pod state is an error.