Kibana does not recognize elasticsearch when the node joins a cluster

Elastic Stack Elasticsearch
1

The elasticsearch version I am using is 2.3.
I have a 2 nodes elasticsearch cluster (with no kibana installed), and an individual elasticsearch data node( with kibana installed). The data node with kibana works perfectly by itself with Kibana showing the local data, but I want the data node with Kibana to join the cluster so that I can visualize data. When I change the settings of Datanode with kibana, Kibana cannot find elasticsearch anymore.

On the datanode_with_kibana node:
When the datanode_with_kibana elasticsearch.yml setting is all commented, and it working individually, and kibana is working perfectly.

But when I set in datanode_with_kibana elasticsearch.yml to join the cluster as follows:

cluster.name: elkstats
network.host: 10.100.16.14

Kibana cannot connect to Elasticsearch

1193×723 34 KB

I checked the cluster status using

curl -XGET 'http://Melasticsearch:9200/_cluster/health?pretty=true'

It shows datanode_with_kibana is recognized as a cluster member as follows:

{
"cluster_name" : "elkstats",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 3,
"number_of_data_nodes" : 3,
"active_primary_shards" : 17,
"active_shards" : 31,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}

The kibana.yml file setting:

server.port: 5601
server.host: "datanode_with_kibana"
elasticsearch.url: "http://10.100.16.14:9200"

Extra information:
The settings of 2 node Elasticsearch cluster is like this:
One Master Node: elasticsearch.yml settings is as follows

cluster.name: elkstats
node.name: Melasticsearch
node.master: true
node.data: true
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["Melasticsearch","datanode1","datanode_with_kibana"]
path.data: /data
network.bind_host: 10.100.16.13
network.publish_host: 10.100.16.13
http.cors.enabled: true
http.cors.allow-origin: "*"

One Data Node: elasticsearch.yml settings is as follows

cluster.name: elkstats
node.name: datanode1
node.master: false
node.data: true
path.data: /data
network.bind_host: 10.100.16.11
network.publish_host: 10.100.16.11
http.cors.enabled: true
http.cors.allow-origin: "*"