Error from server (NotFound): services "quickstart-es-http" not found

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

Hello World!

I'm trying to follow: #### Request Elasticsearch access

yet, running into following error:

$ kubectl get service quickstart-es-http
Error from server (NotFound): services "quickstart-es-http" not found
$
$ kubectl get services
NAME                      TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
kubernetes                ClusterIP   10.37.0.1    <none>        443/TCP    6d10h
quickstart-es             ClusterIP   10.37.4.20   <none>        9200/TCP   4d23h
quickstart-es-discovery   ClusterIP   None         <none>        9300/TCP   4d23h
quickstart-kibana         ClusterIP   10.37.6.26   <none>        5601/TCP   4d23h
$

Please advise.

2

Hi there!

It seems like you are mixing a 0.8.x version of ECK with the latest quickstart instructions.
You can check what version of ECK you are running by running the following command:

kubectl get pods -n elastic-system -o yaml | grep image

If you are running ECK 0.8.x you can either refer to the right guide, 0.8's quickstart, or consider upgrading your ECK installation to its latest release by following 0.9's quickstart guide.

Please note that resources created with 0.8.x are not compatible with 0.9 as noted in 0.9's release highlights, so in your case it would probably be better to delete them before upgrading the operator.

If you're already running ECK 0.9, please refer to 0.9's release highlights for instructions on how to remove incompatible resources created with an older version of ECK (0.8.x) and then follow the 0.9's quickstart guide for creating a new Elasticsearch cluster.

3

Idan,

Thank you for your reply! I guess that is possible that when I started following guide, 0.8 was current at the time and now 0.9 is...

running: 0.9.0

$ kubectl get pods -n elastic-system -o yaml | grep image
      image: docker.elastic.co/eck/eck-operator:0.9.0
      imagePullPolicy: IfNotPresent
      image: docker.elastic.co/eck/eck-operator:0.9.0
      imageID: docker-pullable://docker.elastic.co/eck/eck-operator@sha256:9f134a7647371fc4e627c592496ff1ef1c3d50d6206fa6fb3571aacdb6ade574
$

How do I go about deleting resources created with 0.8, so I can install 0.9?

4

Hi @alexus

kubectl delete elasticsearch <name>

You can get your Elasticsearch deployment name by running kubectl get elasticsearch:

NAME      HEALTH   NODES   VERSION   PHASE         AGE
idan-es   green    1       7.1.0     Operational   11m

If you also deployed kibana, delete it as well: kubectl delete kibana <name>

Then upgrade ECK to the latest version by following the 0.9 quickstart guide.

5

I'm not sure what's wrong, however per your reply I've tried to delete elasticsearch and kibana:

elasticsearch:

$ kubectl get elasticsearch
NAME         HEALTH   NODES   VERSION   PHASE         AGE
quickstart   green    2       7.2.0     Operational   8d
$ kubectl delete elasticsearch quickstart
elasticsearch.elasticsearch.k8s.elastic.co "quickstart" deleted

even though output says deleted when I check in another terminal, I see elasticsearch is still there:

$ kubectl get elasticsearch
NAME         HEALTH   NODES   VERSION   PHASE         AGE
quickstart   green    2       7.2.0     Operational   9d
$

kibana:

$ kubectl get kibana
NAME         HEALTH   NODES   VERSION   AGE
quickstart   red              7.2.0     8d
$ kubectl delete kibana quickstart
kibana.kibana.k8s.elastic.co "quickstart" deleted

I was able to delete pod, yet that didn't help either(

$ kubectl delete pod quickstart-kibana-76d7c6bd48-2qjwb --namespace=default
pod "quickstart-kibana-76d7c6bd48-2qjwb" deleted
$ kubectl get kibana
NAME         HEALTH   NODES   VERSION   AGE
quickstart   red              7.2.0     9d
$

elastic-operator:

$ kubectl get pods --namespace=elastic-system
NAME                 READY   STATUS    RESTARTS   AGE
elastic-operator-0   1/1     Running   0          23h
$

What about elastic-operator?

Thanks in advance)

6

You are probably running into the issue described here: https://www.elastic.co/guide/en/cloud-on-k8s/0.9/release-highlights-0.9.0.html#k8s-breaking_changes

There are in all likelihood some finalizers on the objects preventing deletion:

kubectl patch elasticsearch quickstart --patch '{"metadata": {"finalizers": []}}' --type=merge
kubectl patch kibana quickstart --patch '{"metadata": {"finalizers": []}}' --type=merge
7

after extensive period of time, I've got following message:

error: An error occurred while waiting for the object to be deleted: an error on the server ("unable to decode an event from the watch stream: read tcp X.X.X.X:63875->Y.Y.Y.Y:443: read: operation timed out") has prevented the request from succeedingerror: An error occurred while waiting for the object to be deleted: an error on the server ("unable to decode an event from the watch stream: read tcp X.X.X.X:58830->Y.Y.Y.Y:443: read: operation timed out") has prevented the request from succeeding

Breaking Changes:

$ kubectl patch elasticsearch quickstart --patch '{"metadata": {"finalizers": []}}' --type=merge
elasticsearch.elasticsearch.k8s.elastic.co/quickstart patched
$ kubectl patch kibana quickstart --patch '{"metadata": {"finalizers": []}}' --type=merge
kibana.kibana.k8s.elastic.co/quickstart patched
$

No resources found:

$ kubectl get elasticsearch
No resources found.
$ kubectl get kibana
No resources found.
$

Is it safe to try again to Deploy ECK in your Kubernetes cluster?

8

Once you upgraded to 0.9 you should be good to deploy new clusters.

9

no need for an upgrade, as I am already using 0.9...

although now running into another issue:

$ cat <<EOF | kubectl apply -f -
> apiVersion: elasticsearch.k8s.elastic.co/v1alpha1
> kind: Elasticsearch
> metadata:
>   name: quickstart
> spec:
>   version: 7.2.0
>   nodes:
>   - nodeCount: 1
>     config:
>       node.master: true
>       node.data: true
>       node.ingest: true
> EOF
Error from server (Timeout): error when creating "STDIN": Timeout: request did not complete within requested timeout 30s
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-20T04:49:16Z", GoVersion:"go1.12.6", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.7-gke.8", GitCommit:"7d3d6f113e933ed1b44b78dff4baf649258415e5", GitTreeState:"clean", BuildDate:"2019-06-19T16:37:16Z", GoVersion:"go1.11.5b4", Compiler:"gc", Platform:"linux/amd64"}
$
10

@alexus can you perform many other actions against the API server (e.g. kubectl get pods)? From that error it's not clear that it is an operator issue so much as an issue with the API server not responding.

11

api server is responding, as I was able to execute kubectl version right after that error...

per your reply:

kubectl get pods --all-namespaces=true:

$ kubectl get pods --all-namespaces=true
NAMESPACE             NAME                                                     READY   STATUS    RESTARTS   AGE
elastic-system        elastic-operator-0                                       1/1     Running   0          16m
gitlab-managed-apps   certmanager-cert-manager-6df979599b-xltdm                1/1     Running   0          16m
gitlab-managed-apps   ingress-nginx-ingress-controller-7cf6944677-5ntrp        1/1     Running   0          16m
gitlab-managed-apps   ingress-nginx-ingress-default-backend-7f7bf55777-l744w   1/1     Running   0          16m
gitlab-managed-apps   prometheus-kube-state-metrics-5d5958bc-2w2tm             1/1     Running   0          16m
gitlab-managed-apps   prometheus-prometheus-server-5c476cc89-qscgj             2/2     Running   0          16m
gitlab-managed-apps   runner-gitlab-runner-6947cc549f-fzjr2                    1/1     Running   0          16m
gitlab-managed-apps   tiller-deploy-7cbfdc5df7-9gxkx                           1/1     Running   0          16m
kube-system           calico-node-vertical-autoscaler-579467d76c-khp7m         1/1     Running   2          16m
kube-system           calico-node-x6cpr                                        2/2     Running   0          20h
kube-system           calico-node-xvdjd                                        2/2     Running   0          16m
kube-system           calico-typha-65bfd5544b-hkcjg                            1/1     Running   0          16m
kube-system           calico-typha-horizontal-autoscaler-847fc7bc8d-ff7j4      1/1     Running   0          16m
kube-system           calico-typha-vertical-autoscaler-dc95cc498-t2qcr         1/1     Running   1          16m
kube-system           event-exporter-v0.2.4-5f88c66fb7-gtvpl                   2/2     Running   2          16m
kube-system           fluentd-gcp-scaler-59b7b75cd7-rtvnr                      1/1     Running   0          16m
kube-system           fluentd-gcp-v3.2.0-drrg9                                 2/2     Running   0          16m
kube-system           fluentd-gcp-v3.2.0-lvtl6                                 2/2     Running   0          20h
kube-system           heapster-v1.6.0-beta.1-8b5666cf6-57bgr                   3/3     Running   2          16m
kube-system           ip-masq-agent-2l4fk                                      1/1     Running   0          16m
kube-system           ip-masq-agent-rxw9z                                      1/1     Running   0          20h
kube-system           kube-dns-6987857fdb-457gz                                4/4     Running   0          16m
kube-system           kube-dns-6987857fdb-tlqw2                                4/4     Running   0          20h
kube-system           kube-dns-autoscaler-bb58c6784-7p2cn                      1/1     Running   0          16m
kube-system           kube-proxy-gke-test-default-pool-cbc7ebb8-0z9l           1/1     Running   0          16m
kube-system           kube-proxy-gke-test-default-pool-cbc7ebb8-9c5t           1/1     Running   0          20h
kube-system           l7-default-backend-fd59995cd-7s4t9                       1/1     Running   0          16m
kube-system           metrics-server-v0.3.1-57c75779f-mcwgc                    2/2     Running   0          16m
kube-system           prometheus-to-sd-fxr52                                   1/1     Running   0          16m
kube-system           prometheus-to-sd-tnw7h                                   1/1     Running   0          20h
kube-system           tiller-deploy-58c844889c-2xtcg                           1/1     Running   0          16m
$

elastic-operator:

$ kubectl -n elastic-system logs statefulset.apps/elastic-operator | tail -18
{"level":"info","ts":1565407823.1509497,"logger":"manager","msg":"Setting up webhooks"}
{"level":"info","ts":1565407823.8920846,"logger":"manager","msg":"Starting the manager","uuid":"33b9a574-b8c0-11e9-9788-66c5545c5018","namespace":"elastic-system","version":"0.9.0","build_hash":"8280d41","build_date":"2019-07-29T14:26:01Z","build_snapshot":"false"}
{"level":"info","ts":1565407824.195746,"logger":"kubebuilder.webhook","msg":"installing webhook configuration in cluster"}
{"level":"info","ts":1565407824.1992793,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"apmserver-controller"}
{"level":"info","ts":1565407824.199369,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"apm-es-association-controller"}
{"level":"info","ts":1565407824.1994004,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"kibana-association-controller"}
{"level":"info","ts":1565407824.1994479,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"elasticsearch-controller"}
{"level":"info","ts":1565407824.1994774,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"kibana-controller"}
{"level":"info","ts":1565407824.1995258,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"license-controller"}
{"level":"info","ts":1565407824.199554,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"trial-controller"}
{"level":"info","ts":1565407824.299501,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"kibana-association-controller","worker count":1}
{"level":"info","ts":1565407824.3006725,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"apmserver-controller","worker count":1}
{"level":"info","ts":1565407824.300739,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"apm-es-association-controller","worker count":1}
{"level":"info","ts":1565407824.3007946,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"trial-controller","worker count":1}
{"level":"info","ts":1565407824.3008096,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"elasticsearch-controller","worker count":1}
{"level":"info","ts":1565407824.3008258,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"kibana-controller","worker count":1}
{"level":"info","ts":1565407824.3008394,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"license-controller","worker count":1}
{"level":"info","ts":1565407825.0831032,"logger":"kubebuilder.webhook","msg":"starting the webhook server."}
$
12

bump?

Error from server (Timeout): error when creating "STDIN": Timeout: request did not complete within requested timeout 30s

13

Hey @alexus, do you have access to the API server logs? That might be helpful to determine why it's timing out when you try to do a kubectl apply

15

I had to paste it to pastebin, I hope that's okay)

https://pastebin.com/tNwixUpT

maybe due to following:

Failed calling webhook, failing open [validation.elasticsearch.elastic.co](http://validation.elasticsearch.elastic.co/): failed calling webhook "[validation.elasticsearch.elastic.co](http://validation.elasticsearch.elastic.co/)": Post [https://elastic-webhook-service.elastic-system.svc:443/validate-elasticsearches?timeout=30s](https://elastic-webhook-service.elastic-system.svc/validate-elasticsearches?timeout=30s): net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

16

So it looks like the operator is starting correctly (or at least started correctly once), because the webhook is getting created. But the API server cannot communicate with the webhook. In managed systems where the API server is external to the cluster, or if there are network policies in place that block inter-namespace communication this is expected. For managed vendors they will normally have documentation on how to allow communication.

That said, the current webhook configuration in 0.9.0 should be failing open, and indeed it looks like it is:


W0821 23:05:12.556120       1 dispatcher.go:67] Failed calling webhook, failing open validation.elasticsearch.elastic.co: failed calling webhook "validation.elasticsearch.elastic.co": Post https://elastic-webhook-service.elastic-system.svc:443/validate-elasticsearches?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
E0821 23:05:12.556189       1 dispatcher.go:68] failed calling webhook "validation.elasticsearch.elastic.co": Post https://elastic-webhook-service.elastic-system.svc:443/validate-elasticsearches?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

But it looks like a timeout might still be canceling the request for some reason. I see there are some upstream issues with a similar issue:

But I haven't been able to reproduce it. If you cannot resolve the communication issue between the API server and the operator then it may be worth it to remove the webhook (kubectl delete validatingwebhookconfigurations.admissionregistration.k8s.io validating-webhook-configuration), and configure the operator to not create the webhook automatically (see here).

17

My kubernetes has cluster network policy enabled, and as soon as I disabled it...

$ gcloud container clusters update test --no-enable-network-policy
Enabling/Disabling Network Policy causes a rolling update of all 
cluster nodes, similar to performing a cluster upgrade.  This 
operation is long-running and will block other operations on the 
cluster (including delete) until it has run to completion.

Do you want to continue (Y/n)?  y

Updating test...done.                                                                                                                                                                                                                        
Updated [https://container.googleapis.com/v1/projects/X-X-X/zones/us-east4-a/clusters/test].
To inspect the contents of your cluster, go to: https://console.cloud.google.com/kubernetes/workload_/gcloud/us-east4-a/test?project=X-X-X
$ 
$ cat <<EOF | kubectl apply -f -
> apiVersion: elasticsearch.k8s.elastic.co/v1alpha1
> kind: Elasticsearch
> metadata:
>   name: quickstart
> spec:
>   version: 7.2.0
>   nodes:
>   - nodeCount: 1
>     config:
>       node.master: true
>       node.data: true
>       node.ingest: true
> EOF
elasticsearch.elasticsearch.k8s.elastic.co/quickstart created
$
18

tried one more time and ran into same issue as last time(

19

If I'm understanding correctly, it worked as soon as you disabled the network policy? If so that points to needing to add a network policy allowing the API server to communicate with the webhook server (or as mentioned, disabling the webhook configuration).

20

It certainly appears so, however I then went ahead and killed kubernetes cluster and redid it without cluster network policy enabled and experience same timeout as before.

Thank You and everyone else replies on this topic, I appreciate all help) Since this topic isn't same issue as timeout, I created separate topic and posted before your reply.