Error from server (NotFound): services "quickstart-es-http" not found

Hello World!

I'm trying to follow: #### Request Elasticsearch access

yet, running into following error:

$ kubectl get service quickstart-es-http
Error from server (NotFound): services "quickstart-es-http" not found
$
$ kubectl get services
NAME                      TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)    AGE
kubernetes                ClusterIP   10.37.0.1    <none>        443/TCP    6d10h
quickstart-es             ClusterIP   10.37.4.20   <none>        9200/TCP   4d23h
quickstart-es-discovery   ClusterIP   None         <none>        9300/TCP   4d23h
quickstart-kibana         ClusterIP   10.37.6.26   <none>        5601/TCP   4d23h
$ 

Please advise.

Hi there!

It seems like you are mixing a 0.8.x version of ECK with the latest quickstart instructions.
You can check what version of ECK you are running by running the following command:

kubectl get pods -n elastic-system -o yaml | grep image

If you are running ECK 0.8.x you can either refer to the right guide, 0.8's quickstart, or consider upgrading your ECK installation to its latest release by following 0.9's quickstart guide.

Please note that resources created with 0.8.x are not compatible with 0.9 as noted in 0.9's release highlights, so in your case it would probably be better to delete them before upgrading the operator.

If you're already running ECK 0.9, please refer to 0.9's release highlights for instructions on how to remove incompatible resources created with an older version of ECK (0.8.x) and then follow the 0.9's quickstart guide for creating a new Elasticsearch cluster.

Idan,

Thank you for your reply! I guess that is possible that when I started following guide, 0.8 was current at the time and now 0.9 is...

running: 0.9.0

$ kubectl get pods -n elastic-system -o yaml | grep image
      image: docker.elastic.co/eck/eck-operator:0.9.0
      imagePullPolicy: IfNotPresent
      image: docker.elastic.co/eck/eck-operator:0.9.0
      imageID: docker-pullable://docker.elastic.co/eck/eck-operator@sha256:9f134a7647371fc4e627c592496ff1ef1c3d50d6206fa6fb3571aacdb6ade574
$ 

How do I go about deleting resources created with 0.8, so I can install 0.9?

Hi @alexus

kubectl delete elasticsearch <name>

You can get your Elasticsearch deployment name by running kubectl get elasticsearch:

NAME      HEALTH   NODES   VERSION   PHASE         AGE
idan-es   green    1       7.1.0     Operational   11m

If you also deployed kibana, delete it as well: kubectl delete kibana <name>

Then upgrade ECK to the latest version by following the 0.9 quickstart guide.

I'm not sure what's wrong, however per your reply I've tried to delete elasticsearch and kibana:

elasticsearch:

$ kubectl get elasticsearch
NAME         HEALTH   NODES   VERSION   PHASE         AGE
quickstart   green    2       7.2.0     Operational   8d
$ kubectl delete elasticsearch quickstart
elasticsearch.elasticsearch.k8s.elastic.co "quickstart" deleted

even though output says deleted when I check in another terminal, I see elasticsearch is still there:

$ kubectl get elasticsearch
NAME         HEALTH   NODES   VERSION   PHASE         AGE
quickstart   green    2       7.2.0     Operational   9d
$

kibana:

$ kubectl get kibana
NAME         HEALTH   NODES   VERSION   AGE
quickstart   red              7.2.0     8d
$ kubectl delete kibana quickstart
kibana.kibana.k8s.elastic.co "quickstart" deleted

I was able to delete pod, yet that didn't help either(

$ kubectl delete pod quickstart-kibana-76d7c6bd48-2qjwb --namespace=default
pod "quickstart-kibana-76d7c6bd48-2qjwb" deleted
$ kubectl get kibana
NAME         HEALTH   NODES   VERSION   AGE
quickstart   red              7.2.0     9d
$ 

elastic-operator:

$ kubectl get pods --namespace=elastic-system
NAME                 READY   STATUS    RESTARTS   AGE
elastic-operator-0   1/1     Running   0          23h
$ 

What about elastic-operator?

Thanks in advance)

You are probably running into the issue described here: https://www.elastic.co/guide/en/cloud-on-k8s/0.9/release-highlights-0.9.0.html#k8s-breaking_changes

There are in all likelihood some finalizers on the objects preventing deletion:

kubectl patch elasticsearch quickstart --patch '{"metadata": {"finalizers": []}}' --type=merge
kubectl patch kibana quickstart --patch '{"metadata": {"finalizers": []}}' --type=merge

after extensive period of time, I've got following message:

error: An error occurred while waiting for the object to be deleted: an error on the server ("unable to decode an event from the watch stream: read tcp X.X.X.X:63875->Y.Y.Y.Y:443: read: operation timed out") has prevented the request from succeedingerror: An error occurred while waiting for the object to be deleted: an error on the server ("unable to decode an event from the watch stream: read tcp X.X.X.X:58830->Y.Y.Y.Y:443: read: operation timed out") has prevented the request from succeeding

Breaking Changes:

$ kubectl patch elasticsearch quickstart --patch '{"metadata": {"finalizers": []}}' --type=merge
elasticsearch.elasticsearch.k8s.elastic.co/quickstart patched
$ kubectl patch kibana quickstart --patch '{"metadata": {"finalizers": []}}' --type=merge
kibana.kibana.k8s.elastic.co/quickstart patched
$

No resources found:

$ kubectl get elasticsearch
No resources found.
$ kubectl get kibana
No resources found.
$

Is it safe to try again to Deploy ECK in your Kubernetes cluster?

Once you upgraded to 0.9 you should be good to deploy new clusters.

no need for an upgrade, as I am already using 0.9...

although now running into another issue:

$ cat <<EOF | kubectl apply -f -
> apiVersion: elasticsearch.k8s.elastic.co/v1alpha1
> kind: Elasticsearch
> metadata:
>   name: quickstart
> spec:
>   version: 7.2.0
>   nodes:
>   - nodeCount: 1
>     config:
>       node.master: true
>       node.data: true
>       node.ingest: true
> EOF
Error from server (Timeout): error when creating "STDIN": Timeout: request did not complete within requested timeout 30s
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"15", GitVersion:"v1.15.0", GitCommit:"e8462b5b5dc2584fdcd18e6bcfe9f1e4d970a529", GitTreeState:"clean", BuildDate:"2019-06-20T04:49:16Z", GoVersion:"go1.12.6", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.7-gke.8", GitCommit:"7d3d6f113e933ed1b44b78dff4baf649258415e5", GitTreeState:"clean", BuildDate:"2019-06-19T16:37:16Z", GoVersion:"go1.11.5b4", Compiler:"gc", Platform:"linux/amd64"}
$

@alexus can you perform many other actions against the API server (e.g. kubectl get pods)? From that error it's not clear that it is an operator issue so much as an issue with the API server not responding.

api server is responding, as I was able to execute kubectl version right after that error...

per your reply:

kubectl get pods --all-namespaces=true:

$ kubectl get pods --all-namespaces=true
NAMESPACE             NAME                                                     READY   STATUS    RESTARTS   AGE
elastic-system        elastic-operator-0                                       1/1     Running   0          16m
gitlab-managed-apps   certmanager-cert-manager-6df979599b-xltdm                1/1     Running   0          16m
gitlab-managed-apps   ingress-nginx-ingress-controller-7cf6944677-5ntrp        1/1     Running   0          16m
gitlab-managed-apps   ingress-nginx-ingress-default-backend-7f7bf55777-l744w   1/1     Running   0          16m
gitlab-managed-apps   prometheus-kube-state-metrics-5d5958bc-2w2tm             1/1     Running   0          16m
gitlab-managed-apps   prometheus-prometheus-server-5c476cc89-qscgj             2/2     Running   0          16m
gitlab-managed-apps   runner-gitlab-runner-6947cc549f-fzjr2                    1/1     Running   0          16m
gitlab-managed-apps   tiller-deploy-7cbfdc5df7-9gxkx                           1/1     Running   0          16m
kube-system           calico-node-vertical-autoscaler-579467d76c-khp7m         1/1     Running   2          16m
kube-system           calico-node-x6cpr                                        2/2     Running   0          20h
kube-system           calico-node-xvdjd                                        2/2     Running   0          16m
kube-system           calico-typha-65bfd5544b-hkcjg                            1/1     Running   0          16m
kube-system           calico-typha-horizontal-autoscaler-847fc7bc8d-ff7j4      1/1     Running   0          16m
kube-system           calico-typha-vertical-autoscaler-dc95cc498-t2qcr         1/1     Running   1          16m
kube-system           event-exporter-v0.2.4-5f88c66fb7-gtvpl                   2/2     Running   2          16m
kube-system           fluentd-gcp-scaler-59b7b75cd7-rtvnr                      1/1     Running   0          16m
kube-system           fluentd-gcp-v3.2.0-drrg9                                 2/2     Running   0          16m
kube-system           fluentd-gcp-v3.2.0-lvtl6                                 2/2     Running   0          20h
kube-system           heapster-v1.6.0-beta.1-8b5666cf6-57bgr                   3/3     Running   2          16m
kube-system           ip-masq-agent-2l4fk                                      1/1     Running   0          16m
kube-system           ip-masq-agent-rxw9z                                      1/1     Running   0          20h
kube-system           kube-dns-6987857fdb-457gz                                4/4     Running   0          16m
kube-system           kube-dns-6987857fdb-tlqw2                                4/4     Running   0          20h
kube-system           kube-dns-autoscaler-bb58c6784-7p2cn                      1/1     Running   0          16m
kube-system           kube-proxy-gke-test-default-pool-cbc7ebb8-0z9l           1/1     Running   0          16m
kube-system           kube-proxy-gke-test-default-pool-cbc7ebb8-9c5t           1/1     Running   0          20h
kube-system           l7-default-backend-fd59995cd-7s4t9                       1/1     Running   0          16m
kube-system           metrics-server-v0.3.1-57c75779f-mcwgc                    2/2     Running   0          16m
kube-system           prometheus-to-sd-fxr52                                   1/1     Running   0          16m
kube-system           prometheus-to-sd-tnw7h                                   1/1     Running   0          20h
kube-system           tiller-deploy-58c844889c-2xtcg                           1/1     Running   0          16m
$

elastic-operator:

$ kubectl -n elastic-system logs statefulset.apps/elastic-operator | tail -18
{"level":"info","ts":1565407823.1509497,"logger":"manager","msg":"Setting up webhooks"}
{"level":"info","ts":1565407823.8920846,"logger":"manager","msg":"Starting the manager","uuid":"33b9a574-b8c0-11e9-9788-66c5545c5018","namespace":"elastic-system","version":"0.9.0","build_hash":"8280d41","build_date":"2019-07-29T14:26:01Z","build_snapshot":"false"}
{"level":"info","ts":1565407824.195746,"logger":"kubebuilder.webhook","msg":"installing webhook configuration in cluster"}
{"level":"info","ts":1565407824.1992793,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"apmserver-controller"}
{"level":"info","ts":1565407824.199369,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"apm-es-association-controller"}
{"level":"info","ts":1565407824.1994004,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"kibana-association-controller"}
{"level":"info","ts":1565407824.1994479,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"elasticsearch-controller"}
{"level":"info","ts":1565407824.1994774,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"kibana-controller"}
{"level":"info","ts":1565407824.1995258,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"license-controller"}
{"level":"info","ts":1565407824.199554,"logger":"kubebuilder.controller","msg":"Starting Controller","controller":"trial-controller"}
{"level":"info","ts":1565407824.299501,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"kibana-association-controller","worker count":1}
{"level":"info","ts":1565407824.3006725,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"apmserver-controller","worker count":1}
{"level":"info","ts":1565407824.300739,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"apm-es-association-controller","worker count":1}
{"level":"info","ts":1565407824.3007946,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"trial-controller","worker count":1}
{"level":"info","ts":1565407824.3008096,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"elasticsearch-controller","worker count":1}
{"level":"info","ts":1565407824.3008258,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"kibana-controller","worker count":1}
{"level":"info","ts":1565407824.3008394,"logger":"kubebuilder.controller","msg":"Starting workers","controller":"license-controller","worker count":1}
{"level":"info","ts":1565407825.0831032,"logger":"kubebuilder.webhook","msg":"starting the webhook server."}
$