Quickstart Deploy an Elasticsearch cluster get stuck in unknown Health & ApplyingChanges Phase

scollector65 · February 19, 2021, 5:40am

I just started ECK and I try Quickstart docs.

Deploy ECK in your Kubernetes cluster | Elastic Cloud on Kubernetes [1.4] | Elastic
Deploy an Elasticsearch cluster | Elastic Cloud on Kubernetes [1.4] | Elastic

but my elasticsearch get stuck in unknown Health & ApplyingChanges Phase.
I retry sometimes with uninstall ECK

Uninstall ECK | Elastic Cloud on Kubernetes [1.4] | Elastic

I guess it is network problem. but I don't know how solve this.
Sorry for my bad english and please help me.

> kubectl get elasticsearch

NAME         HEALTH    NODES   VERSION   PHASE             AGE
quickstart   unknown   1       7.11.0    ApplyingChanges   21m

> kubectl describe elasticsearch

Name:         quickstart
Namespace:    default
Labels:       <none>
Annotations:  common.k8s.elastic.co/controller-version: 1.4.0
              elasticsearch.k8s.elastic.co/initial-master-nodes: quickstart-es-default-0
API Version:  elasticsearch.k8s.elastic.co/v1
Kind:         Elasticsearch
Metadata:
  Creation Timestamp:  2021-02-19T05:11:21Z
  Generation:          2
  Resource Version:    64195832
  Self Link:           /apis/elasticsearch.k8s.elastic.co/v1/namespaces/default/elasticsearches/quickstart
  UID:                 4b6a2730-8a45-47ca-bd90-09768045ec9e
Spec:
  Auth:
  Http:
    Service:
      Metadata:
        Creation Timestamp:  <nil>
      Spec:
    Tls:
      Certificate:
  Node Sets:
    Config:
      node.store.allow_mmap:  false
    Count:                    1
    Name:                     default
    Pod Template:
      Metadata:
        Creation Timestamp:  <nil>
      Spec:
        Containers:  <nil>
  Transport:
    Service:
      Metadata:
        Creation Timestamp:  <nil>
      Spec:
    Tls:
      Certificate:
  Update Strategy:
    Change Budget:
  Version:  7.11.0
Status:
  Available Nodes:  1
  Health:           unknown
  Phase:            ApplyingChanges
  Version:          7.11.0
Events:
  Type     Reason      Age                  From                      Message
  ----     ------      ----                 ----                      -------
  Warning  Unexpected  3m48s (x2 over 12m)  elasticsearch-controller  Could not update cluster license: while getting current license level Get "https://quickstart-es-http.default.svc:9200/_license": dial tcp 10.43.26.15:9200: connect: connection timed out
  Warning  Unexpected  3m48s (x2 over 12m)  elasticsearch-controller  Could not update remote clusters in Elasticsearch settings

> kubectl -n elastic-system logs -f statefulset.apps/elastic-operator

...
{"log.level":"info","@timestamp":"2021-02-19T05:22:42.218Z","log.logger":"driver","message":"Could not update cluster license","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","err":"while getting current license level Get \"https://quickstart-es-http.default.svc:9200/_license\": dial tcp 10.43.26.15:9200: connect: connection timed out","namespace":"default","es_name":"quickstart"}
{"log.level":"info","@timestamp":"2021-02-19T05:23:40.978Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","kind":"ConfigMap","namespace":"elastic-system","name":"elastic-licensing"}
{"log.level":"error","@timestamp":"2021-02-19T05:24:49.450Z","log.logger":"driver","message":"Could not update remote clusters in Elasticsearch settings","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","namespace":"default","es_name":"quickstart","error":"Get \"https://quickstart-es-http.default.svc:9200/_cluster/settings\": dial tcp 10.43.26.15:9200: connect: connection timed out","error.stack_trace":"github.com/elastic/cloud-on-k8s/pkg/controller/elasticsearch/driver.(*defaultDriver).Reconcile\n\t/go/src/github.com/elastic/cloud-on-k8s/pkg/controller/elasticsearch/driver/driver.go:214\ngithub.com/elastic/cloud-on-k8s/pkg/controller/elasticsearch.(*ReconcileElasticsearch).internalReconcile\n\t/go/src/github.com/elastic/cloud-on-k8s/pkg/controller/elasticsearch/elasticsearch_controller.go:290\ngithub.com/elastic/cloud-on-k8s/pkg/controller/elasticsearch.(*ReconcileElasticsearch).Reconcile\n\t/go/src/github.com/elastic/cloud-on-k8s/pkg/controller/elasticsearch/elasticsearch_controller.go:199\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:244\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:218\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:197\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.14/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.14/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.14/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.14/pkg/util/wait/wait.go:90"}
{"log.level":"info","@timestamp":"2021-02-19T05:25:40.978Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","kind":"ConfigMap","namespace":"elastic-system","name":"elastic-licensing"}
{"log.level":"info","@timestamp":"2021-02-19T05:26:56.682Z","log.logger":"elasticsearch-uuid","message":"Recoverable error while retrieving Elasticsearch cluster UUID","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","namespace":"default","es_name":"quickstart","error":"Get \"https://quickstart-es-http.default.svc:9200/\": dial tcp 10.43.26.15:9200: connect: connection timed out"}
{"log.level":"info","@timestamp":"2021-02-19T05:27:40.978Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","kind":"ConfigMap","namespace":"elastic-system","name":"elastic-licensing"}
{"log.level":"info","@timestamp":"2021-02-19T05:29:03.914Z","log.logger":"elasticsearch-controller","message":"Ending reconciliation run","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","iteration":18,"namespace":"default","es_name":"quickstart","took":509.15582179}
{"log.level":"error","@timestamp":"2021-02-19T05:29:03.914Z","log.logger":"controller","message":"Reconciler error","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","controller":"elasticsearch-controller","name":"quickstart","namespace":"default","error":"Get \"https://quickstart-es-http.default.svc:9200/_nodes/_master\": dial tcp 10.43.26.15:9200: connect: connection timed out","errorCauses":[{"error":"Get \"https://quickstart-es-http.default.svc:9200/_nodes/_master\": dial tcp 10.43.26.15:9200: connect: connection timed out"}],"error.stack_trace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:246\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:218\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:197\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.14/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.14/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.14/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.14/pkg/util/wait/wait.go:90"}
{"log.level":"info","@timestamp":"2021-02-19T05:29:03.914Z","log.logger":"elasticsearch-controller","message":"Starting reconciliation run","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","iteration":19,"namespace":"default","es_name":"quickstart"}
{"log.level":"info","@timestamp":"2021-02-19T05:29:40.978Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","kind":"ConfigMap","namespace":"elastic-system","name":"elastic-licensing"}
{"log.level":"info","@timestamp":"2021-02-19T05:31:11.402Z","log.logger":"driver","message":"Could not update cluster license","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","err":"while getting current license level Get \"https://quickstart-es-http.default.svc:9200/_license\": dial tcp 10.43.26.15:9200: connect: connection timed out","namespace":"default","es_name":"quickstart"}
{"log.level":"info","@timestamp":"2021-02-19T05:31:40.978Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","kind":"ConfigMap","namespace":"elastic-system","name":"elastic-licensing"}
{"log.level":"error","@timestamp":"2021-02-19T05:33:18.634Z","log.logger":"driver","message":"Could not update remote clusters in Elasticsearch settings","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","namespace":"default","es_name":"quickstart","error":"Get \"https://quickstart-es-http.default.svc:9200/_cluster/settings\": dial tcp 10.43.26.15:9200: connect: connection timed out","error.stack_trace":"github.com/elastic/cloud-on-k8s/pkg/controller/elasticsearch/driver.(*defaultDriver).Reconcile\n\t/go/src/github.com/elastic/cloud-on-k8s/pkg/controller/elasticsearch/driver/driver.go:214\ngithub.com/elastic/cloud-on-k8s/pkg/controller/elasticsearch.(*ReconcileElasticsearch).internalReconcile\n\t/go/src/github.com/elastic/cloud-on-k8s/pkg/controller/elasticsearch/elasticsearch_controller.go:290\ngithub.com/elastic/cloud-on-k8s/pkg/controller/elasticsearch.(*ReconcileElasticsearch).Reconcile\n\t/go/src/github.com/elastic/cloud-on-k8s/pkg/controller/elasticsearch/elasticsearch_controller.go:199\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:244\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:218\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).worker\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/internal/controller/controller.go:197\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.14/pkg/util/wait/wait.go:155\nk8s.io/apimachinery/pkg/util/wait.BackoffUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.14/pkg/util/wait/wait.go:156\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.14/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.18.14/pkg/util/wait/wait.go:90"}
{"log.level":"info","@timestamp":"2021-02-19T05:33:40.978Z","log.logger":"generic-reconciler","message":"Updating resource","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","kind":"ConfigMap","namespace":"elastic-system","name":"elastic-licensing"}

scollector65 · February 21, 2021, 11:28pm

I found error when I deploy ECK.

{"log.level":"error","@timestamp":"2021-02-21T23:20:07.472Z","log.logger":"manager","message":"Timeout elapsed waiting for webhook certificate to be available","service.version":"1.4.0+4aff0b98","service.type":"eck","ecs.version":"1.4.0","path":"/tmp/k8s-webhook-server/serving-certs/tls.crt","timeout_seconds":30,"error":"timed out waiting for the condition","error.stack_trace":"github.com/elastic/cloud-on-k8s/cmd/manager.setupWebhook\n\t/go/src/github.com/elastic/cloud-on-k8s/cmd/manager/main.go:797\ngithub.com/elastic/cloud-on-k8s/cmd/manager.startOperator\n\t/go/src/github.com/elastic/cloud-on-k8s/cmd/manager/main.go:554\ngithub.com/elastic/cloud-on-k8s/cmd/manager.doRun.func2\n\t/go/src/github.com/elastic/cloud-on-k8s/cmd/manager/main.go:321"}

michael.morello · February 22, 2021, 7:21am

Hi, first of all sorry, I mixed up questions

Regarding the last error you mentioned, the one about the webhook, it might happen during the first deployment, but the operator should eventually recover.

Could you share some details about your setup like the K8S distribution and the version, if you are running the operator in a cloud provider ?
Thanks.

system · March 22, 2021, 7:21am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.