I have a six node Elasticsearch cluster with Kibana and Sense installed. Latest versions of everything, cluster initialized today.
The cluster is in Amazon EC2 and I'm accessing Kibana and Elasticsearch via ssh tunnels. I have a load balancer in front of the Elasticsearch cluster and am tunnelling through a host in the same virtual private cloud, so there's no issue with network ACLs or access privileges.
I've researched trying to get Sense to work and have found the http.cors settings. These are set in my elasticsearch.yml file:
Kibana is remapped to localhost:5601
Elasticsearch is remapped to localhost:9200
I have Marvel, Timelion and Sense plugins installed. Marvel is operational and gathering stats.
Here's what works:
I can open and use Kibana on localhost:5601 just fine.
I can use Marvel.
I can load Timelion but I haven't actually tried to use it, because I don't really know how yet.
I can load Sense and it supports typeahead.
Here's what doesn't work:
Any query in Sense against the server http://localhost:9200 fails with this error message:
Error connecting to 'http://localhost:9200/_cluster/health':
Client request error: connect ECONNREFUSED 127.0.0.1:9200
Direct connection to the IP address is not an option; I have to go through the tunnels. It doesn't matter if the ssh tunnels are established using an IP address or the FQDNs of the hosts.
I can use curl to perform queries directly on localhost:9200. Example:
$ curl -XGET http://localhost:9200/_cluster/health?pretty
"cluster_name" : "development-elk-es",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 6,
"number_of_data_nodes" : 6,
"active_primary_shards" : 4,
"active_shards" : 8,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
I've also tried tunneling directly to ports 9200 and 5601 on one of the elasticsearch nodes instead of the load balancer, but the behavior is exactly the same. I've even ssh'ed directly to the Elasticsearch server and tunneled to its own public interface. No dice.
So, I'm at a loss. Any advice would be appreciated.