Misconfigured instance or proxy (302 redirect loop)

Hi!

I am running into issues with my Kibana instance behind an Nginx reverse proxy. I have a very minimal Kibana config which could be an issue though it seems like I have what I need.

That being said, I am quite sure that my Nginx configuration is going to be the issue. Just hoping that someone knowledgeable might be able to point out my error.

I have included my Nginx configuration, Kibana configuration, as well as the output from my Kibana container. Kibana is running inside of Docker, Nginx is not.
Kibana:

---
server.host: "0"
server.name: "kibana"
elasticsearch.hosts: [ "http://localhost:9200" ]
xpack.monitoring.ui.container.elasticsearch.enabled: true

elasticsearch.username: elastic
elasticsearch.password: <password>

Nginx:

server {
    listen 443 ssl;
    server_name kibana.example.gg;

    add_header Strict-Transport-Security "max-age=63072000; includeSubDomains; preload";
    add_header X-Content-Type-Options nosniff;
    add_header Referrer-Policy "same-origin";
    add_header Feature-Policy "geolocation 'none'; microphone 'none'; camera 'none'";
    add_header Content-Security-Policy "default-src 'none'; script-src 'self'; style-src 'self' 'unsafe-inline'; font-src 'self'; connect-src 'self' wss:; img-src 'self' data:; frame-ancestors https://example.gg; base-uri 'self'; form-action 'self'";

    add_header Cache-Control no-cache;
    expires 0;

    error_log /var/log/nginx/error_kibana.log;
    access_log /var/log/nginx/access_kibana.log;

    ssl_certificate /etc/letsencrypt/live/example.gg/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/example.gg/privkey.pem;

    set $proxy_pass_url http://127.0.0.1:5601;

    # Main location which proxy's Kibana backend server
    location / {
        proxy_set_header Host $proxy_pass_url;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_http_version 1.1;
        proxy_set_header Connection "Keep-Alive";
        proxy_set_header Proxy-Connection "Keep-Alive";
        proxy_set_header Authorization "";
        proxy_pass $proxy_pass_url/;
        proxy_redirect $proxy_pass_url/ /;
    }
    # Reverse proxy of assets and front end app
    location ~ (/app|/translations|/node_modules|/built_assets/|/bundles|/es_admin|/plugins|/api|/ui|/elasticsearch|/spaces/enter) {
        proxy_pass          $proxy_pass_url;
        proxy_set_header    Host $host;
        proxy_set_header    X-Real-IP $remote_addr;
        proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header    X-Forwarded-Proto $scheme;
        proxy_set_header    X-Forwarded-Host $http_host;
        proxy_set_header    Authorization "";
        proxy_hide_header   Authorization;
    }
}

server {
    listen 80;
    server_name kibana.example.gg;
    return 302 https://$host$request_uri;
}

Kibana logs:
https://pastebin.com/raw/aExsbbSU

My apologies if this is too much detail or if this should be directed elsewhere, I didn't want to miss anything. Thank you for your time in advance!

Bump..

Still having this issue, replicated it without Docker. This is what I have followed but still no luck https://medium.com/@vvangemert/how-to-set-up-kibana-7-nginx-reverse-proxy-a43b63654b7

@jbpratt from what I'm seeing through the logs, I don't think the problem right now is the reverse proxy itself. It looks like Kibana is not even starting.

Can we try to change the username from elastic to kibana please?

Also, what are the Kibana and the elasticsearch version you are using?
Have you used any other Kibana version with the Elasticsearch nodes you are currently using?
Is the elasticsearch node accessible to the kibana node through localhost:9200 ? I mean, can you successfully run curl -PUT -u elastic:<YOUR_PASSWORD> http://localhost:9200 from within Kibana instance?

My elasticsearch container is running 7.6.0 (image: docker.elastic.co/elasticsearch/elasticsearch:7.6.0). It is a single master node currently, neither Elasticsearch or Kibana was ran below 7.6.0.

Kibana has been updated and moved outside of Docker in the process of debugging (7.6.0 -> 7.6.1)

$ apt show kibana
Package: kibana
Version: 7.6.1

With changing the username from elastic to kibana, I receive the following error:

{"type":"log","@timestamp":"2020-03-18T18:43:11Z","tags":["warning","plugins","licensing"],"pid":21977,"message":"License information could not be obtained from Elasticsearch due to [security_exception] failed to authenticate user [kibana], with { header={ WWW-Authenticate=\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\" } } :: {\"path\":\"/_xpack\",\"statusCode\":401,\"response\":\"{\\\"error\\\":{\\\"root_cause\\\":[{\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"failed to authenticate user [kibana]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}}],\\\"type\\\":\\\"security_exception\\\",\\\"reason\\\":\\\"failed to authenticate user [kibana]\\\",\\\"header\\\":{\\\"WWW-Authenticate\\\":\\\"Basic realm=\\\\\\\"security\\\\\\\" charset=\\\\\\\"UTF-8\\\\\\\"\\\"}},\\\"status\\\":401}\",\"wwwAuthenticateDirective\":\"Basic realm=\\\"security\\\" charset=\\\"UTF-8\\\"\"} error"}

And I can successfully curl against elasticsearch

$ curl -X GET -u elastic:<password> "localhost:9200/" >/dev/null 2>&1; echo $?
0

I am happy to move Kibana back into a container and provide a log dump of starting the stack if you think that would be beneficial. Also, I have since reverted to the default Kibana configuration only adding my credentials.

Thank you for your time!

Edit:
Here is also something that may help

curl -X GET -s -u elastic:<password> "localhost:9200/_cat/indices" | rg kibana
green  open .monitoring-kibana-7-2020.03.12   GxvsuP5xSh-F37i3HXhBGQ 1 0    8640      0   1.9mb   1.9mb
green  open .monitoring-kibana-7-2020.03.13   Rf79P65hSkKeXNUXvvcpbg 1 0    8639      0   1.9mb   1.9mb
green  open .monitoring-kibana-7-2020.03.14   47hKhOFSSxmWZZb9wmFZZw 1 0    8639      0   1.9mb   1.9mb
green  open .monitoring-kibana-7-2020.03.15   INkbz84rTYeYq6q__jv4YQ 1 0    8640      0   1.9mb   1.9mb
green  open .monitoring-kibana-7-2020.03.16   jrRuLn7MS96Zl4YFC2I8sw 1 0    7174      0   1.5mb   1.5mb
green  open .kibana_task_manager_1            oIho8_htTaGMnFEmonbTpg 1 0       2      2  20.6kb  20.6kb
green  open .kibana_1                         4uYOwOccSVaRqPnLxZHxHw 1 0       3      0  15.3kb  15.3kb

Yeah that would be a good idea. Can we move to the original kibana docker image please? Also, do u think we can "reinstall" the elasticsearch node from the beginning?

Also, have u tried to run the curl withtout the >/dev/null 2>&1; echo $? ? I think if you got an error the output would also be 0.

Alright, I'll revert those changes now, and I can use a new volume for the container.

Yeah, running curl against the elasticsearch node has been fine, thats how I have ensured my data has been indexing correctly. Do see the edit in the last message in case it helps.

Edit again:
green open .monitoring-kibana-7-2020.03.18 7jcgSctPTtWZzNRGJzAq2w 1 0 52 0 145.9kb 145.9kb
Here is today's index after restarting the kibana instance a few minute ago

@jbpratt let me know about the results.

Can you also run a curl against localhost:9200/_cat/aliases ?

curl -X GET -s -u elastic:<password> "localhost:9200/_cat/aliases"
.kibana_task_manager .kibana_task_manager_1 - - - -
ilm-history-1        ilm-history-1-000002   - - - true
.security            .security-7            - - - -
.kibana              .kibana_1              - - - -
ilm-history-1        ilm-history-1-000001   - - - false

Alright, once you setup a fresh Kibana and a fresh Elasticsearch search with a new volume, share with the me the logs please.

Attached is the output when doing a fresh start, at least using a new volume for elasticsearch.. output

@jbpratt are you now able to access http://localhost:5601? It looks like Kibana started. However it would have been better if you have started ES before Kibana to avoid so much error output :smiley:

Let me set a dependency and try providing a better output. My apologies!

Edit: it seems I do have the dependency for kibana

  kibana:
    image: docker.elastic.co/kibana/kibana:7.6.0
    volumes:
      - type: bind
        source: ./kibana.yml
        target: /usr/share/kibana/config/kibana.yml
        read_only: true
    network_mode: "host"
    ports:
      - "5601:5601"
    depends_on:
      - elasticsearch

Here is output lacking logstash (sorry)
https://pastebin.com/raw/72q02qKf

with curl -v -I -L -X GET -u elastic:<password> "localhost:5601"
Here is the output

* Rebuilt URL to: localhost:5601/
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 5601 (#0)
* Server auth using Basic with user 'elastic'
> GET / HTTP/1.1
> Host: localhost:5601
> Authorization: Basic XXXX
> User-Agent: curl/7.52.1
> Accept: */*
>
< HTTP/1.1 302 Found
HTTP/1.1 302 Found
< location: /spaces/enter
location: /spaces/enter
< kbn-name: Debian-91-stretch-64-minimal
kbn-name: Debian-91-stretch-64-minimal
< kbn-license-sig: X
kbn-license-sig: X
< kbn-xpack-sig: X
kbn-xpack-sig: X
< cache-control: no-cache
cache-control: no-cache
< content-length: 0
content-length: 0
< Date: Wed, 18 Mar 2020 19:54:17 GMT
Date: Wed, 18 Mar 2020 19:54:17 GMT
< Connection: keep-alive
Connection: keep-alive

<
* Curl_http_done: called premature == 0
* Connection #0 to host localhost left intact
* Issue another request to this URL: 'http://localhost:5601/spaces/enter'
* Found bundle for host localhost: 0x561b6d81a8e0 [can pipeline]
* Re-using existing connection! (#0) with host localhost
* Connected to localhost (127.0.0.1) port 5601 (#0)
* Server auth using Basic with user 'elastic'
> GET /spaces/enter HTTP/1.1
> Host: localhost:5601
> Authorization: Basic XXXX
> User-Agent: curl/7.52.1
> Accept: */*
>
< HTTP/1.1 302 Found
HTTP/1.1 302 Found
< location: /app/kibana
location: /app/kibana
< kbn-name: Debian-91-stretch-64-minimal
kbn-name: Debian-91-stretch-64-minimal
< kbn-license-sig: X
kbn-license-sig: X
< kbn-xpack-sig: X
kbn-xpack-sig: X
< content-type: text/html; charset=utf-8
content-type: text/html; charset=utf-8
< cache-control: no-cache
cache-control: no-cache
< content-length: 0
content-length: 0
< Date: Wed, 18 Mar 2020 19:54:17 GMT
Date: Wed, 18 Mar 2020 19:54:17 GMT
< Connection: keep-alive
Connection: keep-alive

<
* Curl_http_done: called premature == 0
* Connection #0 to host localhost left intact
* Issue another request to this URL: 'http://localhost:5601/app/kibana'
* Found bundle for host localhost: 0x561b6d81a8e0 [can pipeline]
* Re-using existing connection! (#0) with host localhost
* Connected to localhost (127.0.0.1) port 5601 (#0)
* Server auth using Basic with user 'elastic'
> GET /app/kibana HTTP/1.1
> Host: localhost:5601
> Authorization: Basic XXXX
> User-Agent: curl/7.52.1
> Accept: */*
>
< HTTP/1.1 200 OK
HTTP/1.1 200 OK
< content-type: text/html; charset=utf-8
content-type: text/html; charset=utf-8
< content-security-policy: script-src 'unsafe-eval' 'self'; worker-src blob: 'self'; style-src 'unsafe-inline' 'self'
content-security-policy: script-src 'unsafe-eval' 'self'; worker-src blob: 'self'; style-src 'unsafe-inline' 'self'
< kbn-name: Debian-91-stretch-64-minimal
kbn-name: Debian-91-stretch-64-minimal
< kbn-license-sig: X
kbn-license-sig: X
< kbn-xpack-sig: X
kbn-xpack-sig: X
< cache-control: no-cache
cache-control: no-cache
< content-length: 87876
content-length: 87876
< vary: accept-encoding
vary: accept-encoding
< accept-ranges: bytes
accept-ranges: bytes
< Date: Wed, 18 Mar 2020 19:54:17 GMT
Date: Wed, 18 Mar 2020 19:54:17 GMT
< Connection: keep-alive
Connection: keep-alive

<
* Excess found in a non pipelined read: excess = 15882 url = /app/kibana (zero-length body)
* Curl_http_done: called premature == 0
* Connection #0 to host localhost left intact

All of this brings me back to thinking it may be an issue with my reverse proxy configuration. :thinking: