Quickstart Deploy an Elasticsearch cluster get stuck in unknown Health & ApplyingChanges Phase

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

I just started ECK and I try Quickstart docs.

but my elasticsearch get stuck in unknown Health & ApplyingChanges Phase.
I retry sometimes with uninstall ECK

I guess it is network problem. but I don't know how solve this.
Sorry for my bad english and please help me.

> kubectl get elasticsearch

NAME         HEALTH    NODES   VERSION   PHASE             AGE
quickstart   unknown   1       7.11.0    ApplyingChanges   21m

> kubectl describe elasticsearch

Name:         quickstart
Namespace:    default
Labels:       <none>
Annotations:  common.k8s.elastic.co/controller-version: 1.4.0
              elasticsearch.k8s.elastic.co/initial-master-nodes: quickstart-es-default-0
API Version:  elasticsearch.k8s.elastic.co/v1
Kind:         Elasticsearch
Metadata:
  Creation Timestamp:  2021-02-19T05:11:21Z
  Generation:          2
  Resource Version:    64195832
  Self Link:           /apis/elasticsearch.k8s.elastic.co/v1/namespaces/default/elasticsearches/quickstart
  UID:                 4b6a2730-8a45-47ca-bd90-09768045ec9e
Spec:
  Auth:
  Http:
    Service:
      Metadata:
        Creation Timestamp:  <nil>
      Spec:
    Tls:
      Certificate:
  Node Sets:
    Config:
      node.store.allow_mmap:  false
    Count:                    1
    Name:                     default
    Pod Template:
      Metadata:
        Creation Timestamp:  <nil>
      Spec:
        Containers:  <nil>
  Transport:
    Service:
      Metadata:
        Creation Timestamp:  <nil>
      Spec:
    Tls:
      Certificate:
  Update Strategy:
    Change Budget:
  Version:  7.11.0
Status:
  Available Nodes:  1
  Health:           unknown
  Phase:            ApplyingChanges
  Version:          7.11.0
Events:
  Type     Reason      Age                  From                      Message
  ----     ------      ----                 ----                      -------
  Warning  Unexpected  3m48s (x2 over 12m)  elasticsearch-controller  Could not update cluster license: while getting current license level Get "https://quickstart-es-http.default.svc:9200/_license": dial tcp 10.43.26.15:9200: connect: connection timed out
  Warning  Unexpected  3m48s (x2 over 12m)  elasticsearch-controller  Could not update remote clusters in Elasticsearch settings

> kubectl -n elastic-system logs -f statefulset.apps/elastic-operator

...
{"log.level":"info","@timestamp":"2021-02-19T05:22:42.218Z","log.logger":"driver","message":"Could not update cluster license","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","err":"while getting current license level Get \"https://quickstart-es-http.default.svc:9200/_license\": dial tcp 10.43.26.15:9200: connect: connection timed out","namespace":"default","es_name":"quickstart"}
{"log.level":"info","@timestamp":"2021-02-19T05:23:40.978Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","kind":"ConfigMap","namespace":"elastic-system","name":"elastic-licensing"}
{"log.level":"error","@timestamp":"2021-02-19T05:24:49.450Z","log.logger":"driver","message":"Could not update remote clusters in Elasticsearch settings","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","namespace":"default","es_name":"quickstart","error":"Get \"https://quickstart-es-http.default.svc:9200/_cluster/settings\": dial tcp 10.43.26.15:9200: connect: connection timed out","error.stack_trace":"github.com/elastic/cloud-on-k8s/pkg/controller/elasticsearch/driver.(*defaultDriver).Reconcile\n\t/go/src/github.com/elastic/cloud-on-k8s/pkg/controller/elasticsearch/driver/driver.go:214\ngithub.com/elastic/cloud-on-k8s/pkg/controller/elasticsearch.(*ReconcileElasticsearch).internalReconcile\n\t/go/src/github.com/elastic/cloud-on-k8s/pkg/controller/elasticsearch/elasticsearch_controller.go:290\ngithub.com/elastic/cloud-on-k8s/pkg/controller/elasticsearch.(*ReconcileElasticsearch).Reconcile\n\t/go/src/github.com/elastic/cloud-on-k8s/pkg/controller/elasticsearch/elasticsearch_controller.go:199\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:244\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:218\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:197\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.14/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.14/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.14/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.14/pkg/util/wait/wait.go:90"}
{"log.level":"info","@timestamp":"2021-02-19T05:25:40.978Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","kind":"ConfigMap","namespace":"elastic-system","name":"elastic-licensing"}
{"log.level":"info","@timestamp":"2021-02-19T05:26:56.682Z","log.logger":"elasticsearch-uuid","message":"Recoverable error while retrieving Elasticsearch cluster UUID","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","namespace":"default","es_name":"quickstart","error":"Get \"https://quickstart-es-http.default.svc:9200/\": dial tcp 10.43.26.15:9200: connect: connection timed out"}
{"log.level":"info","@timestamp":"2021-02-19T05:27:40.978Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","kind":"ConfigMap","namespace":"elastic-system","name":"elastic-licensing"}
{"log.level":"info","@timestamp":"2021-02-19T05:29:03.914Z","log.logger":"elasticsearch-controller","message":"Ending reconciliation run","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","iteration":18,"namespace":"default","es_name":"quickstart","took":509.15582179}
{"log.level":"error","@timestamp":"2021-02-19T05:29:03.914Z","log.logger":"controller","message":"Reconciler error","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","controller":"elasticsearch-controller","name":"quickstart","namespace":"default","error":"Get \"https://quickstart-es-http.default.svc:9200/_nodes/_master\": dial tcp 10.43.26.15:9200: connect: connection timed out","errorCauses":[{"error":"Get \"https://quickstart-es-http.default.svc:9200/_nodes/_master\": dial tcp 10.43.26.15:9200: connect: connection timed out"}],"error.stack_trace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:246\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:218\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:197\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.14/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.14/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.14/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.14/pkg/util/wait/wait.go:90"}
{"log.level":"info","@timestamp":"2021-02-19T05:29:03.914Z","log.logger":"elasticsearch-controller","message":"Starting reconciliation run","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","iteration":19,"namespace":"default","es_name":"quickstart"}
{"log.level":"info","@timestamp":"2021-02-19T05:29:40.978Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","kind":"ConfigMap","namespace":"elastic-system","name":"elastic-licensing"}
{"log.level":"info","@timestamp":"2021-02-19T05:31:11.402Z","log.logger":"driver","message":"Could not update cluster license","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","err":"while getting current license level Get \"https://quickstart-es-http.default.svc:9200/_license\": dial tcp 10.43.26.15:9200: connect: connection timed out","namespace":"default","es_name":"quickstart"}
{"log.level":"info","@timestamp":"2021-02-19T05:31:40.978Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","kind":"ConfigMap","namespace":"elastic-system","name":"elastic-licensing"}
{"log.level":"error","@timestamp":"2021-02-19T05:33:18.634Z","log.logger":"driver","message":"Could not update remote clusters in Elasticsearch settings","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","namespace":"default","es_name":"quickstart","error":"Get \"https://quickstart-es-http.default.svc:9200/_cluster/settings\": dial tcp 10.43.26.15:9200: connect: connection timed out","error.stack_trace":"github.com/elastic/cloud-on-k8s/pkg/controller/elasticsearch/driver.(*defaultDriver).Reconcile\n\t/go/src/github.com/elastic/cloud-on-k8s/pkg/controller/elasticsearch/driver/driver.go:214\ngithub.com/elastic/cloud-on-k8s/pkg/controller/elasticsearch.(*ReconcileElasticsearch).internalReconcile\n\t/go/src/github.com/elastic/cloud-on-k8s/pkg/controller/elasticsearch/elasticsearch_controller.go:290\ngithub.com/elastic/cloud-on-k8s/pkg/controller/elasticsearch.(*ReconcileElasticsearch).Reconcile\n\t/go/src/github.com/elastic/cloud-on-k8s/pkg/controller/elasticsearch/elasticsearch_controller.go:199\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:244\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:218\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:197\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.14/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.14/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.14/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.14/pkg/util/wait/wait.go:90"}
{"log.level":"info","@timestamp":"2021-02-19T05:33:40.978Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","kind":"ConfigMap","namespace":"elastic-system","name":"elastic-licensing"}
2

I found error when I deploy ECK.

{"log.level":"error","@timestamp":"2021-02-21T23:20:07.472Z","log.logger":"manager","message":"Timeout elapsed waiting for webhook certificate to be available","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","path":"/tmp/k8s-webhook-server/serving-certs/tls.crt","timeout_seconds":30,"error":"timed out waiting for the condition","error.stack_trace":"github.com/elastic/cloud-on-k8s/cmd/manager.setupWebhook\n\t/go/src/github.com/elastic/cloud-on-k8s/cmd/manager/main.go:797\ngithub.com/elastic/cloud-on-k8s/cmd/manager.startOperator\n\t/go/src/github.com/elastic/cloud-on-k8s/cmd/manager/main.go:554\ngithub.com/elastic/cloud-on-k8s/cmd/manager.doRun.func2\n\t/go/src/github.com/elastic/cloud-on-k8s/cmd/manager/main.go:321"}
4

Hi, first of all sorry, I mixed up questions :slight_smile:

Regarding the last error you mentioned, the one about the webhook, it might happen during the first deployment, but the operator should eventually recover.

Could you share some details about your setup like the K8S distribution and the version, if you are running the operator in a cloud provider ?
Thanks.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.