Failed to get API Group-Resources

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

I am trying to deploy Elastic Cloud on kubernetes. When I first tryed i have managed to start elasticsearch on kubernetes cluster. But now for some reason, when i try to deploy same quickstart elasticsearch it never even deploy a pod for it. No services made for it. I think becase of elastic-operator is shutting down at start up.

I have tryed to delete all-in-one.yml and redeploy it, i have tryed to reset the cluster and re deploy it. None of them worked. I am successfully deploying other non Elastic pods. The logs for elastic-operator is in bellow (prettifyed):

    {
   "log.level":"info",
   "@timestamp":"2021-01-19T13:04:41.881Z",
   "log.logger":"manager",
   "message":"maxprocs: Updating GOMAXPROCS=1: determined from CPU quota",
   "service.version":"1.3.1+a0a0a212",
   "service.type":"eck",
   "ecs.version":"1.4.0"
}{
   "log.level":"info",
   "@timestamp":"2021-01-19T13:04:41.882Z",
   "log.logger":"manager",
   "message":"Setting default container registry",
   "service.version":"1.3.1+a0a0a212",
   "service.type":"eck",
   "ecs.version":"1.4.0",
   "container_registry":"docker.elastic.co"
}{
   "log.level":"info",
   "@timestamp":"2021-01-19T13:04:41.882Z",
   "log.logger":"manager",
   "message":"Setting up scheme",
   "service.version":"1.3.1+a0a0a212",
   "service.type":"eck",
   "ecs.version":"1.4.0"
}{
   "log.level":"info",
   "@timestamp":"2021-01-19T13:04:41.887Z",
   "log.logger":"manager",
   "message":"Operator configured to manage all namespaces",
   "service.version":"1.3.1+a0a0a212",
   "service.type":"eck",
   "ecs.version":"1.4.0"
}{
   "log.level":"error",
   "@timestamp":"2021-01-19T13:05:11.888Z",
   "log.logger":"controller-runtime.manager",
   "message":"Failed to get API Group-Resources",
   "service.version":"1.3.1+a0a0a212",
   "service.type":"eck",
   "ecs.version":"1.4.0",
   "error":"Get \"https://10.96.0.1:443/api?timeout=1m0s\": dial tcp 10.96.0.1:443: i/o timeout",
   "error.stack_trace":"sigs.k8s.io/controller-runtime/pkg/manager.New\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.6.3/pkg/manager/manager.go:279\ngithub.com/elastic/cloud-on-k8s/cmd/manager.startOperator\n\t/go/src/github.com/elastic/cloud-on-k8s/cmd/manager/main.go:484\ngithub.com/elastic/cloud-on-k8s/cmd/manager.doRun.func2\n\t/go/src/github.com/elastic/cloud-on-k8s/cmd/manager/main.go:319"
}{
   "log.level":"error",
   "@timestamp":"2021-01-19T13:05:11.888Z",
   "log.logger":"manager",
   "message":"Failed to create controller manager",
   "service.version":"1.3.1+a0a0a212",
   "service.type":"eck",
   "ecs.version":"1.4.0",
   "error":"Get \"https://10.96.0.1:443/api?timeout=1m0s\": dial tcp 10.96.0.1:443: i/o timeout",
   "error.stack_trace":"github.com/elastic/cloud-on-k8s/cmd/manager.startOperator\n\t/go/src/github.com/elastic/cloud-on-k8s/cmd/manager/main.go:486\ngithub.com/elastic/cloud-on-k8s/cmd/manager.doRun.func2\n\t/go/src/github.com/elastic/cloud-on-k8s/cmd/manager/main.go:319"
}{
   "log.level":"error",
   "@timestamp":"2021-01-19T13:05:11.888Z",
   "log.logger":"manager",
   "message":"Shutting down due to error",
   "service.version":"1.3.1+a0a0a212",
   "service.type":"eck",
   "ecs.version":"1.4.0",
   "error":"Get \"https://10.96.0.1:443/api?timeout=1m0s\": dial tcp 10.96.0.1:443: i/o timeout",
   "error.stack_trace":"github.com/elastic/cloud-on-k8s/cmd/manager.doRun\n\t/go/src/github.com/elastic/cloud-on-k8s/cmd/manager/main.go:327\ngithub.com/spf13/cobra.(*Command).execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:842\ngithub.com/spf13/cobra.(*Command).ExecuteC\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:950\ngithub.com/spf13/cobra.(*Command).Execute\n\t/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:887\nmain.main\n\t/go/src/github.com/elastic/cloud-on-k8s/cmd/main.go:30\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:204"
}
2

It looks like the operator has trouble reaching the k8s control plane specifically the k8s apiserver. But I am assuming that the k8s apiserver is healthy because you say you can deploy other Pods successfully. But maybe worth double checking the health of the control plane just to be sure.

Do you have any firewall rules preventing the operator from reaching the control plane? It could also be a k8s NetworkPolicy.

3

Ok thank you for your reply, when i run kubectl get cs i get;

    NAME                 STATUS      MESSAGE                                                                                       ERROR
    scheduler            Unhealthy   Get "http://127.0.0.1:10251/healthz": dial tcp 127.0.0.1:10251: connect: connection refused   
    controller-manager   Unhealthy   Get "http://127.0.0.1:10252/healthz": dial tcp 127.0.0.1:10252: connect: connection refused   
    etcd-0               Healthy     {"health":"true"}

kubectl get --raw='/readyz' gives ok result.

I dont have any firewall rules.

I am using standalone containerd runtime. Yesterday someone from us installed docker in master node. Maybe this caused the issue I dont know. I have deleted docker but this didnt help.

4

Pls share the solution .

5

I'm experiencing the same issue, have you found any solution?