Following the quickstart guide: getting "Kibana server is not ready yet."

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

Hi
I have followed the 2.8.0 quickstart guide to the letter
I have used the own certificate method and the sites are secure, I am happy with that.
I have configured DNS and load balancing, and the pods and services look great

NAMESPACE              NAME                                        READY   STATUS    RESTARTS     AGE
kube-system            hostpath-provisioner-69cd9ff5b8-wcxld       1/1     Running   3 (4d ago)   69d
kubernetes-dashboard   dashboard-metrics-scraper-7bc864c59-lr5j8   1/1     Running   1 (4d ago)   16d
kube-system            calico-kube-controllers-68fd98d89c-n8gsv    1/1     Running   2 (4d ago)   7d21h
metallb-system         speaker-frjss                               1/1     Running   1 (4d ago)   17d
kube-system            coredns-6f5f9b5d74-8sksf                    1/1     Running   2 (4d ago)   69d
kubernetes-dashboard   kubernetes-dashboard-85b98b98bb-jz47b       1/1     Running   2 (4d ago)   16d
ingress                nginx-ingress-microk8s-controller-tdwpk     1/1     Running   2 (4d ago)   69d
kube-system            calico-node-7c95t                           1/1     Running   1 (4d ago)   7d21h
metallb-system         controller-5fd797fbf7-xkl4w                 1/1     Running   2 (4d ago)   17d
elastic-system         elastic-operator-0                          1/1     Running   0            147m
default                quickstart-es-default-0                     1/1     Running   0            100m
default                quickstart-kb-7494df9bf9-mjnm5              1/1     Running   0            36m

and the services

NAMESPACE              NAME                          TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)                  AGE
default                kubernetes                    ClusterIP      10.152.183.1     <none>         443/TCP                  69d
kube-system            kube-dns                      ClusterIP      10.152.183.10    <none>         53/UDP,53/TCP,9153/TCP   69d
metallb-system         webhook-service               ClusterIP      10.152.183.31    <none>         443/TCP                  17d
kubernetes-dashboard   kubernetes-dashboard          LoadBalancer   10.152.183.17    10.200.17.42   443:32282/TCP            16d
kubernetes-dashboard   dashboard-metrics-scraper     ClusterIP      10.152.183.139   <none>         8000/TCP                 16d
elastic-system         elastic-webhook-server        ClusterIP      10.152.183.38    <none>         443/TCP                  141m
default                quickstart-es-transport       ClusterIP      None             <none>         9300/TCP                 93m
default                quickstart-es-internal-http   ClusterIP      10.152.183.81    <none>         9200/TCP                 93m
default                quickstart-es-default         ClusterIP      None             <none>         9200/TCP                 93m
default                quickstart-es-http            LoadBalancer   10.152.183.49    10.200.17.41   9200:31780/TCP           93m
default                quickstart-kb-http            LoadBalancer   10.152.183.82    10.200.17.43   5601:30683/TCP           29m

When I navigate to https://elastic.my-domain:9200
I supply username and password and it looks great

But when I got to https://kibana.my-domain:5601
i get an error saying
"Kibana server is not ready yet"

I know I did not miss a step in the quickstart, what could the issue be?

2

Hi @chadleywilson Welcome to the community.

You are going to need to look at the logs for the kibana pod quickstart-kb-7494df9bf9-mjnm5 and see what the error messages are.

Unfortunately, the "Kibana server is not ready yet" message is pretty generic but usually means that Kibana is not connecting to Elasticsearch, either connectivity, auth/auth or cert issue.

Take a look and share what you find.

Also did you test all the components before you configured the Loadbalancers and DNS?

3

Hi Stephen
Yes DNS / Loadbalancer all working, I think becuase there are other systems on the cluster that are working 100%
I agree that there is an auth issue between Kibana and Elastic and I have no idea how to fix it at the moment.

https://elastic.mydomain:9200/_security/enroll/kibana
{"error":{"root_cause":[{"type":"security_exception","reason":"Enrollment mode is not enabled. Set [xpack.security.enrollment.enabled] to true, in order to use this API."}],"type":"security_exception","reason":"Enrollment mode is not enabled. Set [xpack.security.enrollment.enabled] to true, in order to use this API."},"status":403}

the only problem is I can't work out how to enable it and redeploy from the quickstart documentation.

for example I don't know how to make this into yaml, and then where to put it in the yaml
xpack.security.enrollment.enabled

Many thanks for your answer.

4

You shouldn't be using the enrollment method for Kibana because ECK manually configures Kibana. Enrollment is only made for auto configuration which ECK does not use.

Perhaps my friend @Sunile_Manjee can help he is an ECK Expert.

As I suggested before if you look at the Kibana Logs you should be able to see the issues.

The other suggestion is to tear it down and use ECK from scratch with out the extra Load Balancers etc... make sure it works and then add the LBs etc.

5

as @stephenb suggested, logs is the place to start. Quickstart is great when you don't want to tweak much. You are now playing with certs, its time to roll your own k8s manifest. Take a look at ElasticKonductor and the service yamls: https://github.com/sunileman/ElasticKonductor/tree/main/azure/eck/eck-yamls

This will give you a great ahead start.

6

Thank you
I will give this a try, but please bear in mind for an IT infrastructure engineer of 25 years, this is really not up my alley. And cloud infrastructure done via code and physical metal server infrastructure is not the same thing, you see we never learnt to code the physical infrastructure using these languages, it wasn't required. I can't program, but I can script, but it feels like there is not enough documentation to explain to someone like me how to put for example server.publicBaseUrl into the these yaml files. Which file do I put it in, where in the file does it go? And what should it look like?

These are things I am hoping to figure out. Maybe then I can get this running.

7

server.publicBaseUrl param resides within kibana.yml. if you are deploying kibana via ECK, it would go under the config object

kind: Kibana
...
spec:
 ...
  config:
    server.publicBaseUrl: "http://my.kibana.public.url.com"
...

I've worked with ECK extensively and have never had to modify that parameter.

In #elastickonductor, you'll see that it deploys Kibana pods in conjunction with a Load Balancer (LB). The LB has a pod selector affinity associated with it to ensure traffic is correctly routed to the appropriate pods once the LB is accessed.