Thanks for your reply. I tried as you suggested and here is what I found.
Starting by tearing down all of Elasticsearch and Kibana and building new stacks:
Using a 1 node ES cluster, I set Kibana's elasticsearch.url
to the private IP of that one ES node, bypassing any load balancer. This worked fine.
Tear down and rebuild both stacks.
Using a 3 node ES cluster, I set Kibana's elasticsearch.url
to the private IP of one of my three ES nodes, chosen at random, bypassing any load balancer. This also worked fine.
Tear down and rebuild both stacks.
Using a 3 node ES cluster, I set Kibana's elasticsearch.url
to the Amazon ELB which sits in front of the 3 node ES cluster. This resulted in an error creating the '.kibana' index.
{
"type": "log",
"@timestamp": "2017-11-01T21:40:18Z",
"tags": [
"status",
"plugin:elasticsearch@5.6.3",
"info"
],
"pid": 2760,
"state": "yellow",
"message": "Status changed from yellow to yellow - No existing Kibana index found",
"prevState": "yellow",
"prevMsg": "Waiting for Elasticsearch"
}
{
"type": "log",
"@timestamp": "2017-11-01T21:40:49Z",
"tags": [
"status",
"plugin:elasticsearch@5.6.3",
"error"
],
"pid": 2760,
"state": "red",
"message": "Status changed from yellow to red - Waiting for Kibana index \".kibana\" to come online failed.",
"prevState": "yellow",
"prevMsg": "No existing Kibana index found"
}
However, the '.kibana' index did actually get created. My Elasticsearch logs from a few seconds following the above entry:
[2017-11-01T21:43:36,195][INFO ][o.e.c.m.MetaDataCreateIndexService] [i-030fd4186fc5c5b96] [.kibana] creating index, cause [api], templates [], shards [1]/[1], mappings [_default_, index-pattern, server, visualization, search, timelion-sheet, config, dashboard, url]
curl http://internal-elasticsearch-1317456081.us-east-1.elb.amazonaws.com:9200/_cat/indices?v
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
yellow open .kibana WMs0MrFfSFqJ6VJUac7ykg 1 1 0 0 162b 162b
Since the error seems to happen when I have a load balancer in front of Elasticsearch, do you have any advice for configuring an Amazon ELB to sit in front of ES and serve as my elasticsearch.url
?
Here's the ELB config I have currently:
Since I'm running a 3 node Elasticsearch cluster, my preference is to avoid using any of those individual IPs as my elasticsearch.url
, as any of those 3 could die and be replaced by autoscaling. So my preference is to use an ELB (or any kind of load balancer) as the elasticsearch.url
.
Thanks.