Exposing Kibana behind GKE ingress (UNHEALTHY state)

[lmes3oud]

Hi everyone, I'm trying to expose Kibana behind of a GCE ingress, but the ingress reporting the kibana service as UNHEALTHY while it is healthy and ready. Just note that the healthcheck created by the Ingress is still using the default value HTTP on / using the nodeport. Changing the value in GCP console to HTTPS on /login and Port: 5601 doesn't change anything and the service still reported Unhealthy.
I'm using ECK 1.3.1 and below are my configs. I'm I missing anything? Thank you in advance.

apiVersion: elasticsearch.k8s.elastic.co/v1beta1
kind: Elasticsearch
metadata:
  name: d3m0
spec:
  version: 7.10.1
  nodeSets:
  - name: default
    count: 1
    config:
      node.store.allow_mmap: false
---
apiVersion: kibana.k8s.elastic.co/v1beta1
kind: Kibana
metadata:
  name: d3m0
spec:
  version: 7.10.1
  count: 1
  elasticsearchRef:
    name: d3m0
  podTemplate:
    metadata:
      labels:
        kibana: node
    spec:
      containers:
      - name: kibana
        resources:
          limits:
            memory: 1Gi
            cpu: 1
        readinessProbe:
          httpGet:
            scheme: HTTPS
            path: "/login"
            port: 5601
  http:
    service:
      spec:
        type: NodePort

---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: kibana-ingress
    spec:
      backend:
          serviceName: d3m0-kb-http
          servicePort: 5601

[charith-elastic (Charith Ellawala)]

You probably need to enable TLS between GCLB and Kibana by adding the service.alpha.kubernetes.io/app-protocols annotation to the pod. There is an example of this at:

elastic/cloud-on-k8s

[lmes3oud]

Hi @charith-elastic, thank you for the quick reply. Straight to the point !
That annotation was missing In kibana service definition.

Can you please point any tutorial or example on how to expose kibana service using a subpath in the Ingress. I'm currently having 404 error. Thank you in advance.

[charith-elastic (Charith Ellawala)]

I couldn't find a tutorial so I tried it out myself.

Say you want to expose Kibana under the subpath /kibana. You first need to configure Kibana to be aware of this by setting server.basePath, server.rewriteBasePath and server.publicBaseUrl. Then you need to update the readiness probe so that it uses the new path to /login.

---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: hulk
  labels:
    app: hulk
spec:
  version: 7.11.2
  count: 1
  config:
    server:
      basePath: "/kibana"
      rewriteBasePath: true
      publicBaseUrl: "https://elastic.stack/kibana"
  http:
    service:
      metadata:
        labels:
          app: hulk
        annotations:
          # Enable TLS between GCLB and the application
          cloud.google.com/app-protocols: '{"https":"HTTPS"}'
          service.alpha.kubernetes.io/app-protocols: '{"https":"HTTPS"}'
          # Comment out the following line if you are not using a VPC-native cluster
          cloud.google.com/neg: '{"ingress": true}'
  elasticsearchRef:
    name: hulk
  podTemplate:
    spec:
      containers:
        - name: kibana
          readinessProbe:
            # Override the readiness probe as GCLB reuses it for its own healthchecks
            httpGet:
              scheme: HTTPS
              path: "/kibana/login"
              port: 5601

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: hulk
  labels:
    app: hulk
  annotations:
    # Issue certificates for TLS hosts automatically
    cert-manager.io/cluster-issuer: "selfsigning-issuer"
    # Disable HTTP traffic
    kubernetes.io/ingress.allow-http: "false"
spec:
  tls:
    - hosts: ["elastic.stack"]
      secretName: hulk-kibana-cert
  rules:
    - host: "elastic.stack"
      http:
        paths:
          - path: "/kibana/*"
            pathType: Exact
            backend:
              service:
                name: hulk-kb-http
                port:
                  name: https

[lmes3oud]

Hi @charith-elastic, thank you very much, this is really helpful.
I will create a topic about it and accept the solutions for more visibility.