# Issues with Kibana 6.4.1 behind Nginx Reverse Proxy with SSL

**URL:** <https://discuss.elastic.co/t/issues-with-kibana-6-4-1-behind-nginx-reverse-proxy-with-ssl/168302>\
**Category:** Kibana\
**Created:** [February 13, 2019, 10:15pm UTC](https://discuss.elastic.co/t/issues-with-kibana-6-4-1-behind-nginx-reverse-proxy-with-ssl/168302 "2019-02-13T22:15:15Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![TeraInferno](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/terainferno/32/40302_2.png) [@TeraInferno](https://discuss.elastic.co/u/TeraInferno)\
**Post date:** [February 13, 2019, 10:15pm UTC](https://discuss.elastic.co/t/issues-with-kibana-6-4-1-behind-nginx-reverse-proxy-with-ssl/168302/1 "2019-02-13T22:15:15Z")

</div>

Hello. I have tried several configurations in order to get Kibana 6.4.1 to be served from [https://analytics.com/kibana](https://analytics.com/kibana). I am using Nginx as a reverse proxy and using Ansible for management and deployment. Whenever I try navigating too [https://analytics.com/kibana](https://analytics.com/kibana), I get a 502 Bad Gateway.

I see the following in /var/log/nginx/error.log:

```auto
2019/02/13 22:07:21 [error] 16716#16716: *225 connect() failed (111: Connection refused) while connecting to upstream, client: <Client IP Address>, server: analytics.com, request: "GET /kibana/ HTTP/1.1", upstream: "https://127.0.0.1:5061/", host: "analytics.com"

```

Below is my current configuration:

### Nginx Configuration

nginx.conf

```auto
# For more information on configuration, see:
# * Official English Documentation: http://nginx.org/en/docs/
# * Official Russian Documentation: http://nginx.org/ru/docs/

user root;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

# Load dynamic modules. See /usr/share/nginx/README.dynamic.
include /usr/share/nginx/modules/*.conf;

events {
    worker_connections 1024;
  # 'use epoll;' to enable for Linux 2.6+
  # 'use kqueue;' to enable for FreeBSD, OSX

}

http {
    log_format main '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log /var/log/nginx/access.log main;

    sendfile on;

    # tcp_nopush on;
    # tcp_nodelay on;
    keepalive_timeout 90;
    types_hash_max_size 2048;

    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    # Load modular configuration files from the /etc/nginx/conf.d directory.
    # See http://nginx.org/en/docs/ngx_core_module.html#include
    # for more information.
    include /etc/nginx/conf.d/*.conf;

    index index.html index.htm;

}

```

reverseProxy.conf: located in /etc/nginx/conf.d

```auto
# aws kibana upstream server definition
upstream kibaba_server {
  server localhost:5061 fail_timeout=0;
}

# redirects http from port 80 to https through port 443
server {
  listen 80;
  server_name analytics.com;
  return 301 https://$host$request_uri;
}

# HTTPS configuration
server {
  listen 443;
  server_name analytics.com;
  ssl on;
  ssl_certificate /etc/nginx/ssl/certs/analytics.crt;
  ssl_certificate_key /etc/nginx/ssl/private/analytics.key;
  ssl_session_cache builtin:1000 shared:SSL:10m;

  # ssl_ciphers ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS;

  # nginx configuration for aws kibana
  location /kibana/ {
    auth_basic "Restricted";
    auth_basic_user_file /etc/nginx/conf.d/kibana.htpasswd;

    ## Set headers
    # proxy_set_header Host $http_host;
    # proxy_set_header Host $host;
    # proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    # proxy_set_header X-Forwarded-Proto $scheme;
    # proxy_set_header X-Forwarded-Host $host;
    # proxy_set_header X-NginX-Proxy true; 
    # proxy_set_header Connection "Keep-Alive";
    # proxy_set_header Proxy-Connection "Keep-Alive";
    # proxy_set_header X-Real-IP $remote_addr;
    # proxy_set_header Upgrade $http_upgrade;
    # proxy_set_header Connection 'upgrade';

    ## Set other options
    # proxy_cache_bypass $http_upgrade;
    # proxy_http_version 1.1;
    # proxy_redirect off;
    
    ## Have the rewrite before 'proxy_pass'
    # rewrite ^/kibana/(.*)$ /$1 break;

    ## Proxy to kibana
    proxy_pass https://kibaba_server/;

    ## Have the rewrite after 'proxy_pass'
    # rewrite ^/kibana/(.*)$ /$1 break;
  }
}

```

### Kibana Configuration

kibana.yml.j2:

```auto
# Kibana is served by a back end server. This controls which port to use.
server.port: {{ kibana_server_port }}

# The host to bind the server to.
server.host: {{ kibana_server_host }}

# If you are running kibana behind a proxy, and want to mount it at a path,
# specify that path here. The basePath can't end in a slash.
server.basePath: "/kibana"

# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# This setting was effectively always `false` before Kibana 6.3 and will
# default to `true` starting in Kibana 7.0.
# server.rewriteBasePath: true

# The Elasticsearch instance to use for all your queries.
elasticsearch.url: {{ kibana_elasticsearch_url }}

# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
server.ssl.enabled: true
server.ssl.certificate: {{ nginx_ssl_certificate }}
server.ssl.key: {{ nginx_ssl_certificate_key }}

# Set the path to where you would like the process id file to be created.
# pid.file: /var/run/kibana.pid

# If you would like to send the log output to a file you can set the path below.
# logging.dest: stdout
logging.dest: /var/log/kibana/kibana.log

```

I only included the stuff I have been tweaking and left out the stuff that was still at default values and commented out.

### Ansible variable values:

```auto
kibana_version: "6.4.1"
kibana_server_port: 5601
kibana_server_host: "localhost"
kibana_elasticsearch_url: "http://localhost:9200"
kibana_config_path: "/etc/kibana"

nginx_ssl_certificate: "/etc/nginx/ssl/certs/analytics.crt"
nginx_ssl_certificate_key: "/etc/nginx/ssl/private/analytics.key"

```

### Permissions on SSL files

The SSL Certificate files are owned by root and the _ssl-cert_ group

```auto
tree -pug /etc/nginx/ssl
/etc/nginx/ssl
├── [drwxr-xr-x root ssl-cert] certs
│ └── [-rw-r--r-- root ssl-cert] analytics.crt
└── [drwxr-xr-x root ssl-cert] private
    └── [-rw-r----- root ssl-cert] analytics.key

```

What am I doing wrong? How do I get Kibana to be properly reveresed proxied by Nginx at the path /kibana and using SSL?

---

<div class="post-metadata">

**Author:** ![jbudz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jbudz/32/45922_2.png) [@jbudz](https://discuss.elastic.co/u/jbudz)\
**Post date:** [February 14, 2019, 6:32pm UTC](https://discuss.elastic.co/t/issues-with-kibana-6-4-1-behind-nginx-reverse-proxy-with-ssl/168302/2 "2019-02-14T18:32:50Z")

</div>

On first glance, that all looks correct. I wonder if nginx is starting before Kibana is, can we check the kibana logs and try the playbook again with the service already started? There's a good chance service=started in ansible doesn't correlate with the server listening.

---

<div class="post-metadata">

**Author:** ![TeraInferno](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/terainferno/32/40302_2.png) [@TeraInferno](https://discuss.elastic.co/u/TeraInferno)\
**Post date:** [February 14, 2019, 7:08pm UTC](https://discuss.elastic.co/t/issues-with-kibana-6-4-1-behind-nginx-reverse-proxy-with-ssl/168302/3 "2019-02-14T19:08:09Z")

</div>

I have also tried logging into the serve via SSH and restarting both Kibana and Nginx manually using `sudo systemctl restart kibana/nginx`. No change.

Looking at the kibana logs, I don't see anything that pops out to me.

I can also curl both Kibana and Elastic Search:

```auto
curl localhost:9200
{
  "name" : "elasticsearch-1",
  "cluster_name" : "analytics",
  "cluster_uuid" : "4UaEGYdcStak7MN5U1wJNw",
  "version" : {
    "number" : "6.4.1",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "e36acdb",
    "build_date" : "2018-09-13T22:18:07.696808Z",
    "build_snapshot" : false,
    "lucene_version" : "7.4.0",
    "minimum_wire_compatibility_version" : "5.6.0",
    "minimum_index_compatibility_version" : "5.0.0"
  },
  "tagline" : "You Know, for Search"
}
curl -i localhost:5601
HTTP/1.1 200 OK
kbn-name: kibana
kbn-xpack-sig: badf01422f4f45cc8dd7a65fa7dc7f54
cache-control: no-cache
content-type: text/html; charset=utf-8
content-length: 231
accept-ranges: bytes
Date: Thu, 14 Feb 2019 19:01:00 GMT
Connection: keep-alive

<script>var hashRoute = '/kibana/app/kibana';
var defaultRoute = '/kibana/app/kibana';

var hash = window.location.hash;
if (hash.length) {
  window.location = hashRoute + hash;
} else {
  window.location = defaultRoute;
}</script>%                                                                                                                                                                                                                                     

```

I have been testing various nginx config parameters throughout this. You can see them as commented out in the /kibana/location of the reverseProxy.conf snippet I shared.

My current config is now:

```auto
  # nginx configuration for aws kibana
  location /kibana/ {
    # auth_basic "Restricted";
    # auth_basic_user_file /etc/nginx/conf.d/kibana.htpasswd;

    ## Set headers
    # proxy_set_header Host $http_host;
    proxy_set_header Host $host;
    # proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    # proxy_set_header X-Forwarded-Proto $scheme;
    # proxy_set_header X-Forwarded-Host $host;
    # proxy_set_header X-Forwarded-Port $server_port;
    # proxy_set_header X-NginX-Proxy true; 
    proxy_set_header Connection "Keep-Alive";
    proxy_set_header Proxy-Connection "Keep-Alive";
    # proxy_set_header X-Real-IP $remote_addr;
    # proxy_set_header Upgrade $http_upgrade;
    # proxy_set_header Connection 'upgrade';

    ## Set other options
    # proxy_cache_bypass $http_upgrade;
    proxy_http_version 1.1;
    # proxy_redirect off;
    
    ## Have the rewrite before 'proxy_pass'
    # rewrite ^/kibana/(.*)$ /$1 break;

    ## Proxy to kibana
    proxy_pass http://kibana_server/;
    # proxy_pass http://localhost:5061;

    ## Have the rewrite after 'proxy_pass'
    rewrite ^/kibana/(.*)$ /$1 break;
  }

```

I know there was a spelling error on the upstream server definition that I have since corrected (kibaba -\> kibana). That should not be causing the 502 bad gateway errors.

---

<div class="post-metadata">

**Author:** ![TeraInferno](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/terainferno/32/40302_2.png) [@TeraInferno](https://discuss.elastic.co/u/TeraInferno)\
**Post date:** [February 14, 2019, 8:02pm UTC](https://discuss.elastic.co/t/issues-with-kibana-6-4-1-behind-nginx-reverse-proxy-with-ssl/168302/4 "2019-02-14T20:02:09Z")

</div>

I figured it out. The issue was that the nginx reverse proxy to kibana was using port 5061 instead of 5601.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 14, 2019, 8:02pm UTC](https://discuss.elastic.co/t/issues-with-kibana-6-4-1-behind-nginx-reverse-proxy-with-ssl/168302/5 "2019-03-14T20:02:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.