Unable to configure Elasticsearch on Kubernetes 1.22 using the Quickstart guide (Deploy ECK in your Kubernetes cluster | Elastic Cloud on Kubernetes [1.8] | Elastic). It is in "ApplyingChanges" for the last 14 hours.
$ kubectl get Elasticsearch
NAME HEALTH NODES VERSION PHASE AGE
quickstart unknown 7.16.1 ApplyingChanges 14h
Please help to identify the issues. Please find below logs and other details;
kubectl describe Elasticsearch
Name: quickstart
Namespace: default
Labels:
Annotations: common.k8s.elastic.co/controller-version: 1.8.0
elasticsearch.k8s.elastic.co/initial-master-nodes: quickstart-es-default-0
API Version: elasticsearch.k8s.elastic.co/v1
Kind: Elasticsearch
Metadata:
Creation Timestamp: 2021-12-15T14:21:24Z
Generation: 2
Managed Fields:
API Version: elasticsearch.k8s.elastic.co/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
.:
f:kubectl.kubernetes.io/last-applied-configuration:
f:spec:
.:
f:version:
Manager: kubectl-client-side-apply
Operation: Update
Time: 2021-12-15T14:21:24Z
API Version: elasticsearch.k8s.elastic.co/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:annotations:
f:Elasticsearch.k8s.elastic.co/initial-master-nodes:
f:status:
Manager: elastic-operator
Operation: Update
Time: 2021-12-15T14:21:26Z
API Version: elasticsearch.k8s.elastic.co/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
f:health:
f:phase:
f:version:
Manager: elastic-operator
Operation: Update
Subresource: status
Time: 2021-12-15T14:21:27Z
Resource Version: 3706240
UID: de8bf229-e46e-405c-803b-28cc5bb3ed82
Spec:
Auth:
Http:
Service:
Metadata:
Spec:
Tls:
Certificate:
Monitoring:
Logs:
Metrics:
Node Sets:
Config:
node.store.allow_mmap: false
Count: 1
Name: default
Pod Template:
Metadata:
Creation Timestamp:
Spec:
Containers:
Transport:
Service:
Metadata:
Spec:
Tls:
Certificate:
Update Strategy:
Change Budget:
Version: 7.16.1
Status:
Health: unknown
Phase: ApplyingChanges
Version: 7.16.1
Events:
#####################
$ kubectl get events
LAST SEEN TYPE REASON OBJECT MESSAGE
51s Normal FailedBinding persistentvolumeclaim/data-Elasticsearch-data-0 no persistent volumes available for this claim and no storage class is set
51s Normal FailedBinding persistentvolumeclaim/data-Elasticsearch-data-1 no persistent volumes available for this claim and no storage class is set
51s Normal FailedBinding persistentvolumeclaim/data-Elasticsearch-master-0 no persistent volumes available for this claim and no storage class is set
51s Normal FailedBinding persistentvolumeclaim/data-Elasticsearch-master-1 no persistent volumes available for this claim and no storage class is set
51s Normal FailedBinding persistentvolumeclaim/data-Elasticsearch-master-2 no persistent volumes available for this claim and no storage class is set
51s Normal FailedBinding persistentvolumeclaim/Elasticsearch-data-quickstart-es-default-0 no persistent volumes available for this claim and no storage class is set
51s Normal FailedBinding persistentvolumeclaim/Elasticsearch-master-Elasticsearch-master-0 no persistent volumes available for this claim and no storage class is set
51s Normal FailedBinding persistentvolumeclaim/Elasticsearch-master-Elasticsearch-master-1 no persistent volumes available for this claim and no storage class is set
51s Normal FailedBinding persistentvolumeclaim/Elasticsearch-master-Elasticsearch-master-2 no persistent volumes available for this claim and no storage class is set
26m Warning FailedScheduling pod/quickstart-es-default-0 0/4 nodes are available: 4 pod has unbound immediate PersistentVolumeClaims.
######################
$ kubectl -n elastic-system logs -f statefulset.apps/elastic-operator
...........
{"log.level":"info","@timestamp":"2021-12-16T05:10:59.906Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"1.8.0+4f367c38","service.type":"eck","ecs.version":"1.4.0","kind":"Service","namespace":"default","name":"quickstart-es-transport"}
{"log.level":"info","@timestamp":"2021-12-16T05:10:59.916Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"1.8.0+4f367c38","service.type":"eck","ecs.version":"1.4.0","kind":"Service","namespace":"default","name":"quickstart-es-http"}
{"log.level":"info","@timestamp":"2021-12-16T05:10:59.922Z","log.logger":"transport","message":"Skipping pod because it has no IP yet","service.version":"1.8.0+4f367c38","service.type":"eck","ecs.version":"1.4.0","namespace":"default","pod_name":"quickstart-es-default-0"}
{"log.level":"info","@timestamp":"2021-12-16T05:11:00.291Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"1.8.0+4f367c38","service.type":"eck","ecs.version":"1.4.0","kind":"Service","namespace":"default","name":"quickstart-es-default"}
{"log.level":"info","@timestamp":"2021-12-16T05:11:00.304Z","log.logger":"driver","message":"ES cannot be reached yet, re-queuing","service.version":"1.8.0+4f367c38","service.type":"eck","ecs.version":"1.4.0","namespace":"default","es_name":"quickstart"}
{"log.level":"info","@timestamp":"2021-12-16T05:11:00.304Z","log.logger":"Elasticsearch-controller","message":"Ending reconciliation run","service.version":"1.8.0+4f367c38","service.type":"eck","ecs.version":"1.4.0","iteration":5138,"namespace":"default","es_name":"quickstart","took":0.398478515}
{"log.level":"info","@timestamp":"2021-12-16T05:11:10.305Z","log.logger":"Elasticsearch-controller","message":"Starting reconciliation run","service.version":"1.8.0+4f367c38","service.type":"eck","ecs.version":"1.4.0","iteration":5139,"namespace":"default","es_name":"quickstart"}
{"log.level":"info","@timestamp":"2021-12-16T05:11:10.307Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"1.8.0+4f367c38","service.type":"eck","ecs.version":"1.4.0","kind":"Service","namespace":"default","name":"quickstart-es-transport"}
{"log.level":"info","@timestamp":"2021-12-16T05:11:10.320Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"1.8.0+4f367c38","service.type":"eck","ecs.version":"1.4.0","kind":"Service","namespace":"default","name":"quickstart-es-http"}
{"log.level":"info","@timestamp":"2021-12-16T05:11:10.327Z","log.logger":"transport","message":"Skipping pod because it has no IP yet","service.version":"1.8.0+4f367c38","service.type":"eck","ecs.version":"1.4.0","namespace":"default","pod_name":"quickstart-es-default-0"}
{"log.level":"info","@timestamp":"2021-12-16T05:11:10.689Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"1.8.0+4f367c38","service.type":"eck","ecs.version":"1.4.0","kind":"Service","namespace":"default","name":"quickstart-es-default"}
{"log.level":"info","@timestamp":"2021-12-16T05:11:10.703Z","log.logger":"driver","message":"ES cannot be reached yet, re-queuing","service.version":"1.8.0+4f367c38","service.type":"eck","ecs.version":"1.4.0","namespace":"default","es_name":"quickstart"}
{"log.level":"info","@timestamp":"2021-12-16T05:11:10.704Z","log.logger":"Elasticsearch-controller","message":"Ending reconciliation run","service.version":"1.8.0+4f367c38","service.type":"eck","ecs.version":"1.4.0","iteration":5139,"namespace":"default","es_name":"quickstart","took":0.398342956}
{"log.level":"info","@timestamp":"2021-12-16T05:11:20.704Z","log.logger":"Elasticsearch-controller","message":"Starting reconciliation run","service.version":"1.8.0+4f367c38","service.type":"eck","ecs.version":"1.4.0","iteration":5140,"namespace":"default","es_name":"quickstart"}
{"log.level":"info","@timestamp":"2021-12-16T05:11:20.705Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"1.8.0+4f367c38","service.type":"eck","ecs.version":"1.4.0","kind":"Service","namespace":"default","name":"quickstart-es-transport"}
{"log.level":"info","@timestamp":"2021-12-16T05:11:20.711Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"1.8.0+4f367c38","service.type":"eck","ecs.version":"1.4.0","kind":"Service","namespace":"default","name":"quickstart-es-http"}
{"log.level":"info","@timestamp":"2021-12-16T05:11:20.718Z","log.logger":"transport","message":"Skipping pod because it has no IP yet","service.version":"1.8.0+4f367c38","service.type":"eck","ecs.version":"1.4.0","namespace":"default","pod_name":"quickstart-es-default-0"}
{"log.level":"info","@timestamp":"2021-12-16T05:11:21.086Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"1.8.0+4f367c38","service.type":"eck","ecs.version":"1.4.0","kind":"Service","namespace":"default","name":"quickstart-es-default"}
{"log.level":"info","@timestamp":"2021-12-16T05:11:21.099Z","log.logger":"driver","message":"ES cannot be reached yet, re-queuing","service.version":"1.8.0+4f367c38","service.type":"eck","ecs.version":"1.4.0","namespace":"default","es_name":"quickstart"}
{"log.level":"info","@timestamp":"2021-12-16T05:11:21.099Z","log.logger":"Elasticsearch-controller","message":"Ending reconciliation run","service.version":"1.8.0+4f367c38","service.type":"eck","ecs.version":"1.4.0","iteration":5140,"namespace":"default","es_name":"quickstart","took":0.39456042}