Kibana with ingress getting ERR_TOO_MANY_REDIRECTS 302

rubans · February 16, 2021, 2:47pm

When I try and expose the Kibana UI externally using ingress, getting ERR_TOO_MANY_REDIRECTS 302 using https.

Kibana v 7.8.1
Seems to get into an infinite loop.
Works without any issues with port-forward on http://localhost:5601 though.
Not sure if https makes a difference.

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-efk
namespace: kube-logging-dev
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /
ingress.gcp.kubernetes.io/pre-shared-cert: "safe-dev"
kubernetes.io/ingress.class: "nginx"
nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
rules:

host: monitor.foo.com
http:
paths:
- backend:
  serviceName: kibana
  servicePort: 5601
  path: /*
host: monitor.foo.com
http:
paths:
- backend:
  serviceName: elasticsearch
  servicePort: 9200
  path: /elasticsearch(/|$)(.*) #

Any ideas?

rashmi · February 17, 2021, 4:36pm

I think the solution is to add server.rewriteBasePath: true like this

---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana
spec:
  config:
    server.rewriteBasePath: true

and update the readinessProbe like :

 podTemplate:
    spec:
      containers:
      - name: kibana
        readinessProbe:
          httpGet:
            scheme: HTTP
            path: /kibana
            port: 5601
        env:
        - name: SERVER_BASEPATH
          value: "/kibana"

don't forget with more than 1 replica set the redirect from / kibana to /kibana/login doesn't works.
Hope it helps.

Thanks
Rashmi

rubans · February 17, 2021, 11:44pm

I tried this and at threw FATAL errors about basepath rewrite in logs so I added the server.basePath : \kibana in yaml in addition.
However, it doesn't seem to work still.
Get 404s now instead in kibana logs and looks like requests are coming X-Forward-Host.
As mentioned, http://localhost:5601 previously worked without server base rewrite but behind the ingress, it seems to really get stuck with redirects. Really surprised how complicated Kibana to setup on K8, surely should be simpler than this
Is there a known issue with Kibana and reverse proxies that could cause an issue?

Btw, what did you mean regards to issues around support for multiple replicas, surely if they are behind a Nodeport or L4 LB then it should just work? in this case I'm only planning on using a single replica but would be good to understand why this is an issue.

rashmi · February 18, 2021, 3:53am

@jportner /@azasypkin any ideas here ?
Thanks
Rashmi

azasypkin · February 18, 2021, 7:00am

Hey @rubans ,

Can you try to access Kibana in a private tab to exclude any old-cookie related issues, especially since you used HTTP for that same URL already? If it still doesn't help then we may need to see the HAR to know how your redirect loop looks like exactly.

paths:
    backend:
    serviceName: elasticsearch
    servicePort: 9200
    path: /elasticsearch(/|$)(.*) #

Also if I remember correctly Kibana 7.8 may define/elasticsearch path as well so you'd better use another pass to access ES directly.

rubans · February 18, 2021, 10:28am

HI @azasypkin

I have reverted back the server basepath changes and attached the logs.
Retried on private tab but makes no difference and still an issue.
My ingress now looks like this

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: ingress-efk
namespace: kube-logging-dev
annotations:
nginx.ingress.kubernetes.io/rewrite-target: /$2
nginx.ingress.kubernetes.io/use-regex: "true"
# nginx.ingress.kubernetes.io/rewrite-target: /
ingress.gcp.kubernetes.io/pre-shared-cert: "foo"
kubernetes.io/ingress.class: "nginx"
# kubernetes.io/ingress.allow-http: "true"
nginx.ingress.kubernetes.io/ssl-redirect: "false"
spec:
rules:

host: monitor.foo.com
http:
paths:
- backend:
  serviceName: kibana
  servicePort: 5601
  path: /*
host: monitor.foo.com
http:
paths:
- backend:
  serviceName: elasticsearch
  servicePort: 9200
  path: /elasticsearch(/|$)(.*)

Deployment looks like this
apiVersion: v1
kind: ConfigMap
metadata:
name: kibana-configmap
data:
kibana.yml: |
server.name: kibana
server.host: 0.0.0.0
#server.rewriteBasePath: true
#server.basePath: /kibana
elasticsearch.hosts: [ "http://elasticsearch:9200" ]
monitoring.ui.container.elasticsearch.enabled: true

X-Pack security credentials

elasticsearch.username: elastic
elasticsearch.password: changeme

apiVersion: apps/v1
kind: StatefulSet
metadata:
name: kibana
namespace: kube-logging-dev
labels:
app: kibana
spec:
serviceName: kibana
replicas: 1
selector:
matchLabels:
app: kibana
template:
metadata:
labels:
app: kibana
spec:
nodeSelector:
workloadType: "system"
containers:
- name: kibana
image: gcr.io/foo/kibana/kibana:7.8.1-v2
resources:
limits:
cpu: 1000m
requests:
cpu: 100m
readinessProbe:
httpGet:
scheme: HTTP
path: /
port: 5601
env:
- name: ELASTICSEARCH_LOGGING_SERVICE_HOST
value: "elasticsearch"
- name: ELASTICSEARCH_LOGGING_SERVICE_PORT
value: "9200"
ports:
- containerPort: 5601
name: ui
protocol: TCP
securityContext:
runAsUser: 999
runAsGroup: 999
volumeMounts:
- name: config-volume
mountPath: /usr/share/kibana/config
volumes:
- name: config-volume
configMap:
name: kibana-configmap
items:
- key: kibana.yml
path: kibana.yml
4) The issue with the relative path /elasticsearch is not an issue for now as I'm using local cluster connection and works as verified by Kibana logs.
5) I can't attach any files to the post but let me know how to post the HAR and Kibana logs.

Here's a sample of difference between internal kube proxy health check one and my external request:
{"type":"response","@timestamp":"2021-02-18T09:36:46Z","tags":,"pid":6,"method":"get","statusCode":302,"req":{"url":"/spaces/enter","method":"get","headers":{"host":"192.168.160.33:5601","user-agent":"kube-probe/1.18+","referer":"http://192.168.160.33:5601/","accept-encoding":"gzip","connection":"close"},"remoteAddress":"192.168.194.31","userAgent":"192.168.194.31","referer":"http://192.168.160.33:5601/"},"res":{"statusCode":302,"responseTime":39,"contentLength":9},"message":"GET /spaces/enter 302 39ms - 9.0B"}
{"type":"response","@timestamp":"2021-02-18T09:36:46Z","tags":,"pid":6,"method":"get","statusCode":200,"req":{"url":"/app/kibana","method":"get","headers":{"host":"192.168.160.33:5601","user-agent":"kube-probe/1.18+","referer":"http://192.168.160.33:5601/spaces/enter","accept-encoding":"gzip","connection":"close"},"remoteAddress":"192.168.194.31","userAgent":"192.168.194.31","referer":"http://192.168.160.33:5601/spaces/enter"},"res":{"statusCode":200,"responseTime":80,"contentLength":9},"message":"GET /app/kibana 200 80ms - 9.0B"}
{"type":"response","@timestamp":"2021-02-18T09:36:47Z","tags":,"pid":6,"method":"get","statusCode":302,"req":{"url":"/","method":"get","headers":{"host":"monitor.foo.dev.gcp.cloud.uk","x-request-id":"bd47e538951bfb95f77c38613d2a6224","x-real-ip":"192.168.194.28","x-forwarded-for":"192.168.194.28","x-forwarded-host":"monitor.foo.dev.gcp.cloud.uk","x-forwarded-port":"443","x-forwarded-proto":"https","x-scheme":"https","pragma":"no-cache","cache-control":"no-cache","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.182 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.9","sec-fetch-site":"none","sec-fetch-mode":"navigate","sec-fetch-user":"?1","sec-fetch-dest":"document","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"192.168.161.14","userAgent":"192.168.161.14"},"res":{"statusCode":302,"responseTime":22,"contentLength":9},"message":"GET / 302 22ms - 9.0B"}
{"type":"response","@timestamp":"2021-02-18T09:36:47Z","tags":,"pid":6,"method":"get","statusCode":302,"req":{"url":"/","method":"get","headers":{"host":"monitor.foo.dev.gcp.cloud.uk","x-request-id":"e3ddaeadf1672adac660911103f958ce","x-real-ip":"192.168.194.28","x-forwarded-for":"192.168.194.28","x-forwarded-host":"monitor.foo.dev.gcp.cloud.uk","x-forwarded-port":"443","x-forwarded-proto":"https","x-scheme":"https","pragma":"no-cache","cache-control":"no-cache","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.182 Safari/537.36","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.9","sec-fetch-site":"none","sec-fetch-mode":"navigate","sec-fetch-user":"?1","sec-fetch-dest":"document","accept-encoding":"gzip, deflate, br","accept-language":"en-US,en;q=0.9"},"remoteAddress":"192.168.161.14","userAgent":"192.168.161.14"},"res":{"statusCode":302,"responseTime":22,"contentLength":9},"message":"GET / 302 22ms - 9.0B"}

NB : I have tried to obfuscate any senstive info from the logs
Pls let me know next steps as it makes no sense why this wouldn't work.

azasypkin · February 18, 2021, 1:48pm

The issue with the relative path /elasticsearch is not an issue for now as I'm using local cluster connection and works as verified by Kibana logs.

I guess Dev Tools may not work properly, but that's fine, let's solve one problem at a time.

I can't attach any files to the post but let me know how to post the HAR and Kibana logs.

You can just try to attach to the gist at https://gist.github.com/. Same for the configs, unfortunately these aren't really readable when added as a part of the post at Discuss.

rubans · February 18, 2021, 2:30pm

@azasypkin I have attached and obfuscated here:

gist.github.com

https://gist.github.com/rubans/ea274e78de0d029a109c8ff88ce6690b

HAR

{
  "log": {
    "version": "1.2",
    "creator": {
      "name": "WebInspector",
      "version": "537.36"
    },
    "pages": [],
    "entries": [
      {

This file has been truncated. show original

ingress.yaml

---
 # TO DO : use config maps
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress-efk
  namespace: kube-logging-dev
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.ingress.kubernetes.io/use-regex: "true"

This file has been truncated. show original

kibana.deployment.yaml

apiVersion: v1
kind: Service
metadata:
  annotations:
    cloud.google.com/neg: '{"ingress": true}'
  name: kibana
  namespace: kube-logging-dev
  labels:
    app: kibana
spec:

This file has been truncated. show original

There are more than three files. show original

Pls let me know, as it feels like it should work but not sure why.
As mentioned my setup is nginx reverse proxy https --> nginx ingress --> kibana svc --> kibana pod

system · March 18, 2021, 2:31pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
ERR_TOO_MANY_REDIRECTS 302 with ELK Operator 7.10.0 Kibana	7	1356	January 21, 2021
Elasticsearch + Xpack + Kibana + Docker + Nginx Kibana elastic-stack-security , docker	3	393	July 3, 2021
Got 302 code when curl kibana ingress URL Kibana	3	542	August 10, 2021
502 Bad Gateway Ingress nginx with kibana Kibana	2	1082	April 4, 2023
I am unable to log in to the Kibana dashboard. After providing the credentials, it keeps redirecting me back to the login page Kibana	6	40	October 4, 2024

Kibana with ingress getting ERR_TOO_MANY_REDIRECTS 302

X-Pack security credentials

Related topics