Connect Kibana to Nginx load balancer to Elasticsearch

[whiranpat]

I have set up a nginx docker container to act as a load balancer for the elasticsearch cluster. Each elasticsearch node is within their own container on separate vms. When I curl to the nginx container I can see that they are successfully doing a form of round robin and redirect me to a elasticsearch container. However, when I deploy a kibana docker container onto the same vm with the nginx container and point the elasticsearch URL to the nginx load balancer, kibana would say that "unable to connect to Elasticsearch at http://". Wondering if anyone has any solutions for this?

nginx config file:

# in the 'http' context
#proxy_cache_path /var/cache/nginx/cache keys_zone=elasticsearch:10m inactive=60m;

events {
  worker_connections  4096;  ## Default: 1024
}

http {
  upstream elasticsearch_servers {
      zone elasticsearch_servers 64K;
      server <elasticsearch server0>:9200;
      server <elasticsearch server1>:9200;
      server <elasticsearch server2>:9200;
      server <elasticsearch server3>:9200;
      server <elasticsearch server4>:9200;
      keepalive 15;
  }

  server {
      listen 8080;

      location / {
          proxy_pass http://elasticsearch_servers;
          proxy_http_version 1.1;
          proxy_set_header Connection "Keep-Alive";
          proxy_set_header Proxy-Connection "Keep-Alive";
          proxy_connect_timeout 5s;
          proxy_read_timeout 10s;
      }
  }
}

nginx docker command:
docker run --name nginx-lb -v /docker/volumes/nginx/config/nginx.conf:/etc/nginx/nginx.conf:Z -p 8080:8080 -d nginx

kibana config file:

---
# Default Kibana configuration from kibana-docker.

server.name: kibana.<domain name>
server.host: "0"
elasticsearch.url: "http://kibana.<domain name>:8080"
# elasticsearch.username: elastic
# elasticsearch.password: changeme

xpack.security.enabled: false
xpack.monitoring.ui.container.elasticsearch.enabled: true

kibana docker command:
docker run -d --name kibana --publish 5601:5601 --volume /docker/volumes/kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml:Z docker.elastic.co/kibana/kibana:6.2.2

[Mike.Barretta]

@whiranpat it looks like your elasticsearch.url is set to a Kibana host. That setting should point to an Elasticsearch node, generally a dedicated coordinator node, hopefully not the master or other master-eligable nodes, though it can point to any node in the cluster.

[whiranpat]

@Mike.Barretta The URL that we have is a dedicated VM that is housing the Kibana docker container and a Nginx docker container. The idea we are trying to do is have the Kibana docker container point the Elasticsearch.url to the Nginx docker container which acts like a load balancer to point to the other VMs that are hosting Elasticsearch docker containers.

I saw multiple documents that reference this as a solution to spread the load across the Elasticsearch cluster instead of piling everything onto one node. Even the Nginx site has some reference to this. So I was wondering if there something new in the elastic stack 6 that prevent this from happening?

Confirmation of Nginx container reaching other Elasticsearch containers:
We have use postman and curl to verify that the Nginx container was successfully able to establish communication to the Elasticsearch containers.

[Mike.Barretta]

@whiranpat no, there isn't anything done in v6 that would prevent this all from working.

OK, so have the ES cluster, have an nginx load balancer able to connect to each of the ES nodes, and have Kibana container running on same VM hosting the nginx load balancer.

Since you can verify that nginx-to-ES works, can you verify that you can reach the nginx container from the kibana container? If you launch a shell on the running kinana container, you should be able to try and curl or ping the nginx container:

docker exec -i -t <container_id> /bin/bash
curl <nginx_ip>:9200

If that is working, then I'm a bit more stuck. If you haven't already, check out the kibana on docker docs

[whiranpat]

@Mike.Barretta Thanks for the help. The Kibana container wasn't able to connect to the Nginx container as you mention. This really pointed me to the correct path. I look into docker documentations and notice this "--net=host", I then proceed to add this to the docker container command and it was successful. Thank you for the help.

docker run -d --name kibana --publish 5601:5601 --volume --net=host /docker/volumes/kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml:Z docker.elastic.co/kibana/kibana:6.2.2

[Mike.Barretta]

Glad it's working!