Issues with Kibana 6.4.1 behind Nginx Reverse Proxy with SSL


(Andrew Grossnickle) #1

Hello. I have tried several configurations in order to get Kibana 6.4.1 to be served from https://analytics.com/kibana. I am using Nginx as a reverse proxy and using Ansible for management and deployment. Whenever I try navigating too https://analytics.com/kibana, I get a 502 Bad Gateway.

I see the following in /var/log/nginx/error.log:

2019/02/13 22:07:21 [error] 16716#16716: *225 connect() failed (111: Connection refused) while connecting to upstream, client: <Client IP Address>, server: analytics.com, request: "GET /kibana/ HTTP/1.1", upstream: "https://127.0.0.1:5061/", host: "analytics.com"

Below is my current configuration:

Nginx Configuration

nginx.conf

# For more information on configuration, see:
#   * Official English Documentation: http://nginx.org/en/docs/
#   * Official Russian Documentation: http://nginx.org/ru/docs/

user  root;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

# Load dynamic modules. See /usr/share/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
  # 'use epoll;' to enable for Linux 2.6+
  # 'use kqueue;' to enable for FreeBSD, OSX

}

http {
    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile            on;

    # tcp_nopush          on;
    # tcp_nodelay         on;
    keepalive_timeout   90;
    types_hash_max_size 2048;

    include             /etc/nginx/mime.types;
    default_type        application/octet-stream;

    # Load modular configuration files from the /etc/nginx/conf.d directory.
    # See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.
    include /etc/nginx/conf.d/*.conf;

    index  index.html index.htm;

}

reverseProxy.conf: located in /etc/nginx/conf.d

# aws kibana upstream server definition
upstream kibaba_server {
  server localhost:5061 fail_timeout=0;
}

# redirects http from port 80 to https through port 443
server {
  listen         80;
  server_name    analytics.com;
  return         301 https://$host$request_uri;
}

# HTTPS configuration
server  {
  listen  443;
  server_name  analytics.com;
  ssl on;
  ssl_certificate     /etc/nginx/ssl/certs/analytics.crt;
  ssl_certificate_key /etc/nginx/ssl/private/analytics.key;
  ssl_session_cache   builtin:1000  shared:SSL:10m;

  # ssl_ciphers         ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS;

  # nginx configuration for aws kibana
  location /kibana/ {
    auth_basic "Restricted";
    auth_basic_user_file /etc/nginx/conf.d/kibana.htpasswd;

    ## Set headers
    # proxy_set_header      Host                $http_host;
    # proxy_set_header      Host                $host;
    # proxy_set_header      X-Forwarded-For     $proxy_add_x_forwarded_for;
    # proxy_set_header      X-Forwarded-Proto   $scheme;
    # proxy_set_header      X-Forwarded-Host    $host;
    # proxy_set_header      X-NginX-Proxy       true; 
    # proxy_set_header      Connection          "Keep-Alive";
    # proxy_set_header      Proxy-Connection    "Keep-Alive";
    # proxy_set_header      X-Real-IP           $remote_addr;
    # proxy_set_header      Upgrade             $http_upgrade;
    # proxy_set_header      Connection          'upgrade';

    ## Set other options
    # proxy_cache_bypass    $http_upgrade;
    # proxy_http_version    1.1;
    # proxy_redirect        off;
    
    ## Have the rewrite before 'proxy_pass'
    # rewrite ^/kibana/(.*)$ /$1 break;

    ## Proxy to kibana
    proxy_pass            https://kibaba_server/;

    ## Have the rewrite after 'proxy_pass'
    # rewrite ^/kibana/(.*)$ /$1 break;
  }
}

Kibana Configuration

kibana.yml.j2:

# Kibana is served by a back end server. This controls which port to use.
server.port: {{ kibana_server_port }}

# The host to bind the server to.
server.host: {{ kibana_server_host }}

# If you are running kibana behind a proxy, and want to mount it at a path,
# specify that path here. The basePath can't end in a slash.
server.basePath: "/kibana"

# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# This setting was effectively always `false` before Kibana 6.3 and will
# default to `true` starting in Kibana 7.0.
# server.rewriteBasePath: true

# The Elasticsearch instance to use for all your queries.
elasticsearch.url: {{ kibana_elasticsearch_url }}


# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
server.ssl.enabled: true
server.ssl.certificate: {{ nginx_ssl_certificate }}
server.ssl.key: {{ nginx_ssl_certificate_key }}


# Set the path to where you would like the process id file to be created.
# pid.file: /var/run/kibana.pid

# If you would like to send the log output to a file you can set the path below.
# logging.dest: stdout
logging.dest: /var/log/kibana/kibana.log

I only included the stuff I have been tweaking and left out the stuff that was still at default values and commented out.

Ansible variable values:

kibana_version: "6.4.1"
kibana_server_port: 5601
kibana_server_host: "localhost"
kibana_elasticsearch_url: "http://localhost:9200"
kibana_config_path: "/etc/kibana"

nginx_ssl_certificate: "/etc/nginx/ssl/certs/analytics.crt"
nginx_ssl_certificate_key: "/etc/nginx/ssl/private/analytics.key"

Permissions on SSL files

The SSL Certificate files are owned by root and the ssl-cert group

tree -pug /etc/nginx/ssl
/etc/nginx/ssl
├── [drwxr-xr-x root     ssl-cert]  certs
│   └── [-rw-r--r-- root     ssl-cert] analytics.crt
└── [drwxr-xr-x root     ssl-cert]  private
    └── [-rw-r----- root     ssl-cert] analytics.key

What am I doing wrong? How do I get Kibana to be properly reveresed proxied by Nginx at the path /kibana and using SSL?


(Jon Budzenski) #2

On first glance, that all looks correct. I wonder if nginx is starting before Kibana is, can we check the kibana logs and try the playbook again with the service already started? There's a good chance service=started in ansible doesn't correlate with the server listening.


(Andrew Grossnickle) #3

I have also tried logging into the serve via SSH and restarting both Kibana and Nginx manually using sudo systemctl restart kibana/nginx. No change.

Looking at the kibana logs, I don't see anything that pops out to me.

I can also curl both Kibana and Elastic Search:

curl localhost:9200
{
  "name" : "elasticsearch-1",
  "cluster_name" : "analytics",
  "cluster_uuid" : "4UaEGYdcStak7MN5U1wJNw",
  "version" : {
    "number" : "6.4.1",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "e36acdb",
    "build_date" : "2018-09-13T22:18:07.696808Z",
    "build_snapshot" : false,
    "lucene_version" : "7.4.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}
curl -i localhost:5601
HTTP/1.1 200 OK
kbn-name: kibana
kbn-xpack-sig: badf01422f4f45cc8dd7a65fa7dc7f54
cache-control: no-cache
content-type: text/html; charset=utf-8
content-length: 231
accept-ranges: bytes
Date: Thu, 14 Feb 2019 19:01:00 GMT
Connection: keep-alive

<script>var hashRoute = '/kibana/app/kibana';
var defaultRoute = '/kibana/app/kibana';

var hash = window.location.hash;
if (hash.length) {
  window.location = hashRoute + hash;
} else {
  window.location = defaultRoute;
}</script>%                                                                                                                                                                                                                                     

I have been testing various nginx config parameters throughout this. You can see them as commented out in the /kibana/location of the reverseProxy.conf snippet I shared.

My current config is now:

  # nginx configuration for aws kibana
  location /kibana/ {
    # auth_basic "Restricted";
    # auth_basic_user_file /etc/nginx/conf.d/kibana.htpasswd;

    ## Set headers
    # proxy_set_header      Host                $http_host;
    proxy_set_header      Host                $host;
    # proxy_set_header      X-Forwarded-For     $proxy_add_x_forwarded_for;
    # proxy_set_header      X-Forwarded-Proto   $scheme;
    # proxy_set_header      X-Forwarded-Host    $host;
    # proxy_set_header      X-Forwarded-Port    $server_port;
    # proxy_set_header      X-NginX-Proxy       true; 
    proxy_set_header      Connection          "Keep-Alive";
    proxy_set_header      Proxy-Connection    "Keep-Alive";
    # proxy_set_header      X-Real-IP           $remote_addr;
    # proxy_set_header      Upgrade             $http_upgrade;
    # proxy_set_header      Connection          'upgrade';

    ## Set other options
    # proxy_cache_bypass    $http_upgrade;
    proxy_http_version    1.1;
    # proxy_redirect        off;
    
    ## Have the rewrite before 'proxy_pass'
    # rewrite ^/kibana/(.*)$ /$1 break;

    ## Proxy to kibana
    proxy_pass            http://kibana_server/;
    # proxy_pass            http://localhost:5061;

    ## Have the rewrite after 'proxy_pass'
    rewrite ^/kibana/(.*)$ /$1 break;
  }

I know there was a spelling error on the upstream server definition that I have since corrected (kibaba -> kibana). That should not be causing the 502 bad gateway errors.


(Andrew Grossnickle) #4

I figured it out. The issue was that the nginx reverse proxy to kibana was using port 5061 instead of 5601.


(system) closed #5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.