Kibana with ingress getting ERR_TOO_MANY_REDIRECTS 302

When I try and expose the Kibana UI externally using ingress, getting ERR_TOO_MANY_REDIRECTS 302 using https.

Kibana v 7.8.1
Seems to get into an infinite loop.
Works without any issues with port-forward on http://localhost:5601 though.
Not sure if https makes a difference.

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-efk
namespace: kube-logging-dev
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
ingress.gcp.kubernetes.io/pre-shared-cert: "safe-dev"
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
rules:

  • host: monitor.foo.com
    http:
    paths:
    • backend:
      serviceName: kibana
      servicePort: 5601
      path: /*
  • host: monitor.foo.com
    http:
    paths:
    • backend:
      serviceName: elasticsearch
      servicePort: 9200
      path: /elasticsearch(/|$)(.*) #

Any ideas?

I think the solution is to add server.rewriteBasePath: true like this

---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
spec:
  config:
    server.rewriteBasePath: true

and update the readinessProbe like :

 podTemplate:
    spec:
      containers:
      - name: kibana
        readinessProbe:
          httpGet:
            scheme: HTTP
            path: /kibana
            port: 5601
        env:
        - name: SERVER_BASEPATH
          value: "/kibana"

don't forget with more than 1 replica set the redirect from / kibana to /kibana/login doesn't works.
Hope it helps.

Thanks
Rashmi

I tried this and at threw FATAL errors about basepath rewrite in logs so I added the server.basePath : \kibana in yaml in addition.
However, it doesn't seem to work still.
Get 404s now instead in kibana logs and looks like requests are coming X-Forward-Host.
As mentioned, http://localhost:5601 previously worked without server base rewrite but behind the ingress, it seems to really get stuck with redirects. Really surprised how complicated Kibana to setup on K8, surely should be simpler than this :slight_smile:
Is there a known issue with Kibana and reverse proxies that could cause an issue?

  • Btw, what did you mean regards to issues around support for multiple replicas, surely if they are behind a Nodeport or L4 LB then it should just work? in this case I'm only planning on using a single replica but would be good to understand why this is an issue.

@jportner /@azasypkin any ideas here ?
Thanks
Rashmi

Hey @rubans ,

Can you try to access Kibana in a private tab to exclude any old-cookie related issues, especially since you used HTTP for that same URL already? If it still doesn't help then we may need to see the HAR to know how your redirect loop looks like exactly.

paths:
    backend:
    serviceName: elasticsearch
    servicePort: 9200
    path: /elasticsearch(/|$)(.*) #

Also if I remember correctly Kibana 7.8 may define/elasticsearch path as well so you'd better use another pass to access ES directly.

HI @azasypkin

  1. I have reverted back the server basepath changes and attached the logs.
    Retried on private tab but makes no difference and still an issue.
  2. My ingress now looks like this

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-efk
namespace: kube-logging-dev
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$2
nginx.ingress.kubernetes.io/use-regex: "true"
# nginx.ingress.kubernetes.io/rewrite-target: /
ingress.gcp.kubernetes.io/pre-shared-cert: "foo"
kubernetes.io/ingress.class: "nginx"
# kubernetes.io/ingress.allow-http: "true"
nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
rules:

  • host: monitor.foo.com
    http:
    paths:
    • backend:
      serviceName: kibana
      servicePort: 5601
      path: /*
  • host: monitor.foo.com
    http:
    paths:
    • backend:
      serviceName: elasticsearch
      servicePort: 9200
      path: /elasticsearch(/|$)(.*)
  1. Deployment looks like this
    apiVersion: v1
    kind: ConfigMap
    metadata:
    name: kibana-configmap
    data:
    kibana.yml: |
    server.name: kibana
    server.host: 0.0.0.0
    #server.rewriteBasePath: true
    #server.basePath: /kibana
    elasticsearch.hosts: [ "http://elasticsearch:9200" ]
    monitoring.ui.container.elasticsearch.enabled: true

    X-Pack security credentials

    elasticsearch.username: elastic
    elasticsearch.password: changeme


apiVersion: apps/v1
kind: StatefulSet
metadata:
name: kibana
namespace: kube-logging-dev
labels:
app: kibana
spec:
serviceName: kibana
replicas: 1
selector:
matchLabels:
app: kibana
template:
metadata:
labels:
app: kibana
spec:
nodeSelector:
workloadType: "system"
containers:
- name: kibana
image: gcr.io/foo/kibana/kibana:7.8.1-v2
resources:
limits:
cpu: 1000m
requests:
cpu: 100m
readinessProbe:
httpGet:
scheme: HTTP
path: /
port: 5601
env:
- name: ELASTICSEARCH_LOGGING_SERVICE_HOST
value: "elasticsearch"
- name: ELASTICSEARCH_LOGGING_SERVICE_PORT
value: "9200"
ports:
- containerPort: 5601
name: ui
protocol: TCP
securityContext:
runAsUser: 999
runAsGroup: 999
volumeMounts:
- name: config-volume
mountPath: /usr/share/kibana/config
volumes:
- name: config-volume
configMap:
name: kibana-configmap
items:
- key: kibana.yml
path: kibana.yml
4) The issue with the relative path /elasticsearch is not an issue for now as I'm using local cluster connection and works as verified by Kibana logs.
5) I can't attach any files to the post but let me know how to post the HAR and Kibana logs.

Here's a sample of difference between internal kube proxy health check one and my external request:
{"type":"response","@timestamp":"2021-02-18T09:36:46Z","tags":,"pid":6,"method":"get","statusCode":302,"req":{"url":"/spaces/enter","method":"get","headers":{"host":"192.168.160.33:5601","user-agent":"kube-probe/1.18+","referer":"http://192.168.160.33:5601/","accept-encoding":"gzip","connection":"close"},"remoteAddress":"192.168.194.31","userAgent":"192.168.194.31","referer":"http://192.168.160.33:5601/"},"res":{"statusCode":302,"responseTime":39,"contentLength":9},"message":"GET /spaces/enter 302 39ms - 9.0B"}
{"type":"response","@timestamp":"2021-02-18T09:36:46Z","tags":,"pid":6,"method":"get","statusCode":200,"req":{"url":"/app/kibana","method":"get","headers":{"host":"192.168.160.33:5601","user-agent":"kube-probe/1.18+","referer":"http://192.168.160.33:5601/spaces/enter","accept-encoding":"gzip","connection":"close"},"remoteAddress":"192.168.194.31","userAgent":"192.168.194.31","referer":"http://192.168.160.33:5601/spaces/enter"},"res":{"statusCode":200,"responseTime":80,"contentLength":9},"message":"GET /app/kibana 200 80ms - 9.0B"}
{"type":"response","@timestamp":"2021-02-18T09:36:47Z","tags":,"pid":6,"method":"get","statusCode":302,"req":{"url":"/","method":"get","headers":{"host":"monitor.foo.dev.gcp.cloud.uk","x-request-id":"bd47e538951bfb95f77c38613d2a6224","x-real-ip":"192.168.194.28","x-forwarded-for":"192.168.194.28","x-forwarded-host":"monitor.foo.dev.gcp.cloud.uk","x-forwarded-port":"443","x-forwarded-proto":"https","x-scheme":"https","pragma":"no-cache","cache-control":"no-cache","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.182 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.9","sec-fetch-site":"none","sec-fetch-mode":"navigate","sec-fetch-user":"?1","sec-fetch-dest":"document","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"192.168.161.14","userAgent":"192.168.161.14"},"res":{"statusCode":302,"responseTime":22,"contentLength":9},"message":"GET / 302 22ms - 9.0B"}
{"type":"response","@timestamp":"2021-02-18T09:36:47Z","tags":,"pid":6,"method":"get","statusCode":302,"req":{"url":"/","method":"get","headers":{"host":"monitor.foo.dev.gcp.cloud.uk","x-request-id":"e3ddaeadf1672adac660911103f958ce","x-real-ip":"192.168.194.28","x-forwarded-for":"192.168.194.28","x-forwarded-host":"monitor.foo.dev.gcp.cloud.uk","x-forwarded-port":"443","x-forwarded-proto":"https","x-scheme":"https","pragma":"no-cache","cache-control":"no-cache","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.182 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.9","sec-fetch-site":"none","sec-fetch-mode":"navigate","sec-fetch-user":"?1","sec-fetch-dest":"document","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"192.168.161.14","userAgent":"192.168.161.14"},"res":{"statusCode":302,"responseTime":22,"contentLength":9},"message":"GET / 302 22ms - 9.0B"}

NB : I have tried to obfuscate any senstive info from the logs
Pls let me know next steps as it makes no sense why this wouldn't work.

  1. The issue with the relative path /elasticsearch is not an issue for now as I'm using local cluster connection and works as verified by Kibana logs.

I guess Dev Tools may not work properly, but that's fine, let's solve one problem at a time.

  1. I can't attach any files to the post but let me know how to post the HAR and Kibana logs.

You can just try to attach to the gist at https://gist.github.com/. Same for the configs, unfortunately these aren't really readable when added as a part of the post at Discuss.

@azasypkin I have attached and obfuscated here:

Pls let me know, as it feels like it should work but not sure why.
As mentioned my setup is nginx reverse proxy https --> nginx ingress --> kibana svc --> kibana pod