ECK: Kibana via Ingress NGINX Controller 502 Bad Gateway

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

I am having an issue using the latest images of ECK's Elastic + Kibana via Helm Chart, installed with following the quick start guides.

My Kibana pod is in a running state, as is my Elastic pod. The services are all running as well.

There are no errors in the logs to share.

This is what I am seeing:

kubectl get all -n elastic-stack
NAME                                                  READY   STATUS    RESTARTS   AGE
pod/elasticsearch-es-default-0                        1/1     Running   0          6d2h
pod/es-kb-quickstart-eck-kibana-kb-7c48f78499-fmsvz   1/1     Running   0          6d2h

NAME                                          TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/elasticsearch-es-default              ClusterIP   None           <none>        9200/TCP   6d2h
service/elasticsearch-es-http                 ClusterIP   10.43.55.145   <none>        9200/TCP   6d2h
service/elasticsearch-es-internal-http        ClusterIP   10.43.99.193   <none>        9200/TCP   6d2h
service/elasticsearch-es-transport            ClusterIP   None           <none>        9300/TCP   6d2h
service/es-kb-quickstart-eck-kibana-kb-http   ClusterIP   10.43.21.65    <none>        5601/TCP   6d2h

NAME                                             READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/es-kb-quickstart-eck-kibana-kb   1/1     1            1           6d2h

NAME                                                        DESIRED   CURRENT   READY   AGE
replicaset.apps/es-kb-quickstart-eck-kibana-kb-7c48f78499   1         1         1       6d2h

NAME                                        READY   AGE
statefulset.apps/elasticsearch-es-default   1/1     6d2h

kubectl get all -n elastic-system
NAME                     READY   STATUS    RESTARTS         AGE
pod/elastic-operator-0   1/1     Running   23 (2d12h ago)   13d

NAME                               TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)   AGE
service/elastic-operator-webhook   ClusterIP   10.43.75.75   <none>        443/TCP   13d

NAME                                READY   AGE
statefulset.apps/elastic-operator   1/1     13d

kubectl get ing -n elastic-stack -o yaml
apiVersion: v1
items:
- apiVersion: networking.k8s.io/v1
  kind: Ingress
  metadata:
    annotations:
      field.cattle.io/publicEndpoints: '[{"addresses":["10.1.0.66","10.1.0.71","10.1.0.95","10.1.2.216"],"port":443,"protocol":"HTTPS","serviceName":"elastic-stack:es-kb-quickstart-eck-kibana-kb-http","ingressName":"elastic-stack:kibana-ingress","hostname":"redacted.example.com","path":"/kibana/","allNodes":false}]'
      kubectl.kubernetes.io/last-applied-configuration: |
        {"apiVersion":"networking.k8s.io/v1","kind":"Ingress","metadata":{"annotations":{"nginx.ingress.kubernetes.io/add-base-url":"true","nginx.ingress.kubernetes.io/ssl-redirect":"false"},"name":"kibana-ingress","namespace":"elastic-stack"},"spec":{"ingressClassName":"nginx","rules":[{"host":"redacted.example.com","http":{"paths":[{"backend":{"service":{"name":"es-kb-quickstart-eck-kibana-kb-http","port":{"number":5601}}},"path":"/kibana/","pathType":"Prefix"}]}}]}}
      nginx.ingress.kubernetes.io/add-base-url: "true"
      nginx.ingress.kubernetes.io/ssl-redirect: "false"
    creationTimestamp: "2024-12-10T17:14:02Z"
    generation: 2
    name: kibana-ingress
    namespace: elastic-stack
    resourceVersion: "210737293"
    uid: 6db13847-c89c-410b-ae3f-199ba2aa6734
  spec:
    ingressClassName: nginx
    rules:
    - host: redacted.example.com
      http:
        paths:
        - backend:
            service:
              name: es-kb-quickstart-eck-kibana-kb-http
              port:
                number: 5601
          path: /kibana/
          pathType: Prefix
    tls:
    - hosts:
      - redacted.example.com
      secretName: redacted-tls
  status:
    loadBalancer:
      ingress:
      - ip: 10.1.0.66
      - ip: 10.1.0.71
      - ip: 10.1.0.95
      - ip: 10.1.2.216
kind: List
metadata:
  resourceVersion: ""
2

Update, here are logs from the nginx-ingress controller when trying to access the URL:

2024/12/17 13:58:38 [error] 5657#5657: *68176685 upstream prematurely closed connection while reading response header from upstream, client: 10.10.0.104, server: [redacted.example.com](http://redacted.example.com/), request: "GET /kibana/ HTTP/2.0", upstream: "http://10.42.3.78:5601/kibana/", host: "[redacted.example.com](http://redacted.example.com/)"
2024/12/17 13:58:38 [error] 5657#5657: *68176685 upstream prematurely closed connection while reading response header from upstream, client: 10.10.0.104, server: [redacted.example.com](http://redacted.example.com/), request: "GET /kibana/ HTTP/2.0", upstream: "http://10.42.3.78:5601/kibana/", host: "[redacted.example.com](http://redacted.example.com/)"
2024/12/17 13:58:38 [error] 5657#5657: *68176685 upstream prematurely closed connection while reading response header from upstream, client: 10.10.0.104, server: [redacted.example.com](http://redacted.example.com/), request: "GET /kibana/ HTTP/2.0", upstream: "http://10.42.3.78:5601/kibana/", host: "[redacted.example.com](http://redacted.example.com/)"
10.10.0.104 - - [17/Dec/2024:13:58:38 +0000] "GET /kibana/ HTTP/2.0" 502 552 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/[131.0.0.0](http://131.0.0.0/) Safari/537.36" 1048 0.010 [elastic-stack-es-kb-quickstart-eck-kibana-kb-http-5601] [] [10.42.3.78:5601](http://10.42.3.78:5601/), [10.42.3.78:5601](http://10.42.3.78:5601/), [10.42.3.78:5601](http://10.42.3.78:5601/) 0, 0, 0 0.004, 0.003, 0.003 502, 502, 502 024f105c92b7a80c0b4f43454f63923e