Kibana short URL not working behind reverse proxy

I’ve a Kibana running on port 5601. And we have set-up Nginx reverse proxy on port 80 that uses RO credentials and sends requests to kibana on port 5601. This is so that RO (read-only) customers can access dashboard without having the need to enter password

This if I launch Kibana like http://kibana_url:80 or http://kibana_url then it goes via the Nginx and I won’t need to enter password.

If I launch Kibana like http://kibana_url:5601, then it takes me to login page and I need to enter password.

When I access Kibana using the 5601 port and create a shortened URL and share it, the shared shortened URL works fine when it is launched.

But when I access Kibana using http://kibana_url which is via Nginx and create a shortened URL and share it, the shared URL returns page not found.

Snippet of nginx config file:

server {
        listen       80 default_server;
        listen       [::]:80 default_server;
        server_name  node00.foo.bar.com;
        #root         /usr/share/nginx/html;
 
        # Load configuration files for the default server block.
        #include /etc/nginx/default.d/*.conf;
 
        location / {
          proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                #proxy_cookie_path / /;
                #proxy_set_header   Cookie $http_cookie;
                proxy_set_header Authorization "Basic foobarfoobarfoobarfoobarfoobarfoob==";
                proxy_pass http://node00.foo.bar.com:5601;
        }
 
        error_page 404 /404.html;
            location = /40x.html {
        }
 
        error_page 500 502 503 504 /50x.html;
            location = /50x.html {
        }
    }

Except for the short url, everything else works fine when accessed via Nginx Reverse Proxy i.e. http://kibana_url.

Also, here's a snippet of Kibana.log when this error occurs:

{"type":"response","@timestamp":"2020-06-15T09:42:14Z","tags":[],"pid":1950,"method":"get","statusCode":302,"req":{"url":"/goto/abracadabraabracadabraabracadbra","method":"get","headers":{"host":"foo.bar.com","x-real-ip":"A.A.A.A","x-forwarded-for":"A.A.A.A","connection":"close","accept":"text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8","upgrade-insecure-requests":"1","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X F_F_F) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1 Safari/605.1.15","accept-language":"en-us","accept-encoding":"gzip, deflate"},"remoteAddress":"B.B.B.B","userAgent":"B.B.B.B"},"res":{"statusCode":302,"responseTime":23,"contentLength":9},"message":"GET /goto/abracadabraabracadabraabracadbra 302 23ms - 9.0B"}
{"type":"response","@timestamp":"2020-06-15T09:42:15Z","tags":[],"pid":1950,"method":"get","statusCode":404,"req":{"url":"/nginx-logo.png","method":"get","headers":{"host":"foo.bar.com","x-real-ip":"A.A.A.A","x-forwarded-for":"A.A.A.A","connection":"close","accept":"image/png,image/svg+xml,image/*;q=0.8,video/*;q=0.8,*/*;q=0.5","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X F_F_F) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1 Safari/605.1.15","accept-language":"en-us","referer":"http://foo.bar.com/goto/abracadabraabracadabraabracadbra","accept-encoding":"gzip, deflate"},"remoteAddress":"B.B.B.B","userAgent":"B.B.B.B","referer":"http://foo.bar.com/goto/abracadabraabracadabraabracadbra"},"res":{"statusCode":404,"responseTime":52,"contentLength":9},"message":"GET /nginx-logo.png 404 52ms - 9.0B"}
{"type":"response","@timestamp":"2020-06-15T09:42:15Z","tags":[],"pid":1950,"method":"get","statusCode":404,"req":{"url":"/poweredby.png","method":"get","headers":{"host":"foo.bar.com","x-real-ip":"A.A.A.A","x-forwarded-for":"A.A.A.A","connection":"close","accept":"image/png,image/svg+xml,image/*;q=0.8,video/*;q=0.8,*/*;q=0.5","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X F_F_F) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1 Safari/605.1.15","accept-language":"en-us","referer":"http://foo.bar.com/goto/abracadabraabracadabraabracadbra","accept-encoding":"gzip, deflate"},"remoteAddress":"B.B.B.B","userAgent":"B.B.B.B","referer":"http://foo.bar.com/goto/abracadabraabracadabraabracadbra"},"res":{"statusCode":404,"responseTime":52,"contentLength":9},"message":"GET /poweredby.png 404 52ms - 9.0B"}

It seems that ES returns 302 but then Nginx is unable to redirect.

ELK Stack Version: 6.8.6.

Will appreciate some pointers.

-Sandeep

Was able to fix this issue by adding proxy_buffers to nginx.conf file.

In Nginx logs, I could see the following error:

2020/06/15 08:37:32 [error] 2213#0: *111259 upstream sent too big header while reading response header from upstream, client: 1.1.1.1, server: node00.foo.bar.com, request: "GET /goto/abracadabraabracadabraabracadabr HTTP/1.1", upstream: "http://2.2.2.2:5601/goto/abracadabraabracadabraabracadabr", host: "node00.foo.bar.com"

I also found that the short url is working for a small visualization but not for a huge dashboard. This correlates with what the error says above that the data chunk is huge.

So, following this SO link, I added the following to my nginx.conf under location block:

    proxy_buffer_size          128k;
    proxy_buffers              4 256k;
    proxy_busy_buffers_size    256k;

Restarted Nginx and all works fine now. The goto link works.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.