Load balancer between Kibana and Elasticsearch?

Elastic Stack Kibana
1

Kibana/Elasticsearch Alpha 3

Question, I've got two Elasticsearch Client Only nodes running under an Elastic Load Balancer in AWS.. I am able to query the API fine using the ELB as an endpoint, so I know the load balancer works..

If I try to have Kibana target that same LB with elasticsearch.url in kibana.yml, Kibana gives me errors. See attachment. If I just use the IP of a single ES node for elasticsearch.url, Kibana works fine..

What considerations am I missing when trying to get Kibana to talk to ES though an LB?

Screen Shot 2016-05-31 at 4.37.03 PM.png2564×1852 388 KB

2

Can you try calling the GET /_xpack Elasticsearch API from the Kibana server? Try calling it on against each ES node directly and also against the ELB. Let us know if one of those returns a response and other don't or if none of them return a response.

3

It looks like I get the same response whether I curl the API directly, or through the Load Balancer..

[root@ip-10-40-10-118 ~]# curl -XGET -u [creds] http://10.40.65.124:9200/_xpack?pretty=true
{
  "build" : {
    "hash" : "c3f3904",
    "date" : "2016-05-26T08:28:22.447Z"
  },
  "license" : {
    "uid" : "5152e04b-f08f-4876-b7a7-43f8e8b28c1d",
    "type" : "trial",
    "mode" : "trial",
    "status" : "active",
    "expiry_date_in_millis" : 1466362369618
  },
  "features" : {
    "graph" : {
      "description" : "Graph Data Exploration for the Elastic Stack",
      "available" : true,
      "enabled" : true
    },
    "monitoring" : {
      "description" : "Monitoring for the Elastic Stack",
      "available" : true,
      "enabled" : true
    },
    "security" : {
      "description" : "Security for the Elastic Stack",
      "available" : true,
      "enabled" : true
    },
    "watcher" : {
      "description" : "Alerting, Notification and Automation for the Elastic Stack",
      "available" : true,
      "enabled" : true
    }
  },
  "tagline" : "You know, for X"
}
[root@ip-10-40-10-118 ~]# curl -XGET -u [creds] http://internal-elkprod-elastic-lb-url.us-east-1.elb.amazonaws.com:9200/_xpack?pretty=true
{
  "build" : {
    "hash" : "c3f3904",
    "date" : "2016-05-26T08:28:22.447Z"
  },
  "license" : {
    "uid" : "5152e04b-f08f-4876-b7a7-43f8e8b28c1d",
    "type" : "trial",
    "mode" : "trial",
    "status" : "active",
    "expiry_date_in_millis" : 1466362369618
  },
  "features" : {
    "graph" : {
      "description" : "Graph Data Exploration for the Elastic Stack",
      "available" : true,
      "enabled" : true
    },
    "monitoring" : {
      "description" : "Monitoring for the Elastic Stack",
      "available" : true,
      "enabled" : true
    },
    "security" : {
      "description" : "Security for the Elastic Stack",
      "available" : true,
      "enabled" : true
    },
    "watcher" : {
      "description" : "Alerting, Notification and Automation for the Elastic Stack",
      "available" : true,
      "enabled" : true
    }
  },
  "tagline" : "You know, for X"
}
4

Hmm. That's good but also doesn't help us much in resolving this. Do you mind pasting the output from the Kibana server log (in the failure scenario)?

6

There is too much log data to be pasted here.. Here are a few events from when we connect to kibana while elasticsearch.url is set to the load balancer..

Let me know if your looking for something in particular, or if there's a better way to send the log data to you.

{"type":"response","@timestamp":"2016-06-02T22:29:04+00:00","tags":[],"pid":6296,"method":"get","statusCode":304,"req":{"url":"/plugins/graph/icon.png","method":"get","headers":{"host":"localhost:5601","connection":"close","cache-control":"max-age=0","accept":"image/webp,image/*,*/*;q=0.8","if-modified-since":"Tue, 31 May 2016 20:02:41 GMT","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36","referer":"https://10.40.10.118/app/monitoring","accept-encoding":"gzip, deflate, sdch, br","accept-language":"en-US,en;q=0.8","cookie":"user={\"username\":\"esadmin\",\"roles\":[\"superuser\"],\"full_name\":null,\"email\":null,\"metadata\":{}}; sid=Fe26.2**64148af83f3ab3ba500c847153825f18699406cf76d50f5e11ffc6809f63904f*3YAellaksImwYPPj3dCo6Q*s-r-Ggg5GNC2BGtQGySqTlElxuXCEJXvW4IVYaQfbiJikiC89MpzO8U-pbJl-SrSXTtpMX4A0HXDjiWsBtpovoYOCpj2W0qlv6ubGhgReCkEJ63wjhB-sdewYXOR7vZy**e9333a2fe9d83970bcd94d2caa3018fb58eeabe346d94edf2bb1780de15779a5*Nq6AVRFBuJoKcnprMDF1ZH9wYfFUekMI7PWiKC_Y7mU"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"https://10.40.10.118/app/monitoring"},"res":{"statusCode":304,"responseTime":3,"contentLength":9},"message":"GET /plugins/graph/icon.png 304 3ms - 9.0B"}
{"type":"response","@timestamp":"2016-06-02T22:29:04+00:00","tags":[],"pid":6296,"method":"get","statusCode":200,"req":{"url":"/bundles/src/ui/public/images/kibana.svg","method":"get","headers":{"host":"localhost:5601","connection":"close","accept":"image/webp,image/*,*/*;q=0.8","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36","referer":"https://10.40.10.118/bundles/commons.style.css?v=12439","accept-encoding":"gzip, deflate, sdch, br","accept-language":"en-US,en;q=0.8","cookie":"user={\"username\":\"esadmin\",\"roles\":[\"superuser\"],\"full_name\":null,\"email\":null,\"metadata\":{}}; sid=Fe26.2**64148af83f3ab3ba500c847153825f18699406cf76d50f5e11ffc6809f63904f*3YAellaksImwYPPj3dCo6Q*s-r-Ggg5GNC2BGtQGySqTlElxuXCEJXvW4IVYaQfbiJikiC89MpzO8U-pbJl-SrSXTtpMX4A0HXDjiWsBtpovoYOCpj2W0qlv6ubGhgReCkEJ63wjhB-sdewYXOR7vZy**e9333a2fe9d83970bcd94d2caa3018fb58eeabe346d94edf2bb1780de15779a5*Nq6AVRFBuJoKcnprMDF1ZH9wYfFUekMI7PWiKC_Y7mU"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"https://10.40.10.118/bundles/commons.style.css?v=12439"},"res":{"statusCode":200,"responseTime":4,"contentLength":9},"message":"GET /bundles/src/ui/public/images/kibana.svg 200 4ms - 9.0B"}
{"type":"response","@timestamp":"2016-06-02T22:29:04+00:00","tags":[],"pid":6296,"method":"get","statusCode":200,"req":{"url":"/plugins/monitoring/monitoring.svg","method":"get","headers":{"host":"localhost:5601","connection":"close","accept":"image/webp,image/*,*/*;q=0.8","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.102 Safari/537.36","referer":"https://10.40.10.118/app/monitoring","accept-encoding":"gzip, deflate, sdch, br","accept-language":"en-US,en;q=0.8","cookie":"user={\"username\":\"esadmin\",\"roles\":[\"superuser\"],\"full_name\":null,\"email\":null,\"metadata\":{}}; sid=Fe26.2**64148af83f3ab3ba500c847153825f18699406cf76d50f5e11ffc6809f63904f*3YAellaksImwYPPj3dCo6Q*s-r-Ggg5GNC2BGtQGySqTlElxuXCEJXvW4IVYaQfbiJikiC89MpzO8U-pbJl-SrSXTtpMX4A0HXDjiWsBtpovoYOCpj2W0qlv6ubGhgReCkEJ63wjhB-sdewYXOR7vZy**e9333a2fe9d83970bcd94d2caa3018fb58eeabe346d94edf2bb1780de15779a5*Nq6AVRFBuJoKcnprMDF1ZH9wYfFUekMI7PWiKC_Y7mU"},"remoteAddress":"127.0.0.1","userAgent":"127.0.0.1","referer":"https://10.40.10.118/app/monitoring"},"res":{"statusCode":200,"responseTime":6,"contentLength":9},"message":"GET /plugins/monitoring/monitoring.svg 200 6ms - 9.0B"}