# Can't see the second node whilst attempting to build a cluster
**URL:** https://discuss.elastic.co/t/cant-see-the-second-node-whilst-attempting-to-build-a-cluster/163187
**Category:** Elasticsearch
**Tags:** elastic-stack-monitoring
**Created:** [January 7, 2019, 11:08am UTC](https://discuss.elastic.co/t/cant-see-the-second-node-whilst-attempting-to-build-a-cluster/163187 "2019-01-07T11:08:34Z")
**Posts on this page:** 16
**Page:** 1
### Author:  [@sc1](https://discuss.elastic.co/u/sc1)
#### Post date: [January 7, 2019, 11:08am UTC](https://discuss.elastic.co/t/cant-see-the-second-node-whilst-attempting-to-build-a-cluster/163187/1 "2019-01-07T11:08:34Z")
Hello,
I am looking to add a second node and create a cluster. I have installed version 6.5.1 of Elasticsearch on 2 servers. The primary node has the full stack (Elasticsearch, Kibana, Logstash) and the second node has Elasticsearch installed.
**Primary node IP: 192.168.0.64 & Secondary node IP: 192.168.0.139**
I have looked at the following questions: [Example](https://discuss.elastic.co/t/i-can-t-add-a-second-node-to-my-cluster/142223) and haven't been able to find a solution.
There is currently no errors from the logs and everything should be working fine but via Kibana I can't see the secondary node, at all. Networking wise, the two hosts are able to communicate and both configs are near enough identical. The cluster name is identical as well.
In the Discovery section of the YML file:
**Primary Node**
```
discovery.zen.ping.unicast.hosts: ["192.168.0.64","192.168.0.139"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
#discovery.zen.minimum_master_nodes: 2
#
# For more information, consult the zen discovery module documentation.
```
**Secondary Node**
```
discovery.zen.ping.unicast.hosts: ["192.168.0.64","192.168.0.139"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
#discovery.zen.minimum_master_nodes:
#
# For more information, consult the zen discovery module documentation.
#
```
On the Primary Node, when I have uncommented **discovery.zen.minimum\_master\_nodes** it actually doesn't start, I have been looking at the documentation but can't find.
Could someone help me understand why the services are running fine, but I can't seem to get the cluster working as only 1 node is appearing in Kibana.
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [January 7, 2019, 12:44pm UTC](https://discuss.elastic.co/t/cant-see-the-second-node-whilst-attempting-to-build-a-cluster/163187/2 "2019-01-07T12:44:44Z")
You must set `discovery.zen.minimum_master_nodes: 2` on both nodes in order for this to work correctly. However, you say that you tried this and "it actually doesn't start". What do you mean? What did the log messages say?
---
### Author:  [@sc1](https://discuss.elastic.co/u/sc1)
#### Post date: [January 7, 2019, 1:19pm UTC](https://discuss.elastic.co/t/cant-see-the-second-node-whilst-attempting-to-build-a-cluster/163187/3 "2019-01-07T13:19:12Z")
Thanks for the swift response @DavidTurner
I have set the discovery.zen.minimum\_master\_nodes to 2 as suggested. When I do this however the following starts:
```
root@main_node:~# sudo service kibana status
● kibana.service - Kibana
Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2019-01-07 12:56:32 GMT; 1min 21s ago
Main PID: 2886 (node)
Tasks: 10
Memory: 279.4M
CPU: 6.440s
CGroup: /system.slice/kibana.service
└─2886 /usr/share/kibana/bin/../node/bin/node --no-warnings /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml
Jan 07 12:57:46 main_node kibana[2886]: exception] Security must be explicitly enabled when using a trial license. Enable security by setting [xpack.security.enabled] to [true] in the elasticsearch.yml file and restart the node."}
Jan 07 12:57:46 main_node kibana[2886]: {"type":"response","@timestamp":"2019-01-07T12:57:46Z","tags":[],"pid":2886,"method":"get","statusCode":500,"req":{"url":"/favicon.ico","method":"get","headers":{"connection":"upgrade","host":"main_node","pragm
Jan 07 12:57:47 main_node kibana[2886]: {"type":"log","@timestamp":"2019-01-07T12:57:47Z","tags":["info","authentication"],"pid":2886,"message":"Authentication attempt failed: [exception] Security must be explicitly enabled when using a trial license. Enable security
Jan 07 12:57:47 main_node kibana[2886]: {"type":"error","@timestamp":"2019-01-07T12:57:47Z","tags":[],"pid":2886,"level":"error","error":{"message":"[exception] Security must be explicitly enabled when using a trial license. Enable security by setting [xpack.security
Jan 07 12:57:47 main_node kibana[2886]: eption] Security must be explicitly enabled when using a trial license. Enable security by setting [xpack.security.enabled] to [true] in the elasticsearch.yml file and restart the node."}
Jan 07 12:57:47 main_node kibana[2886]: {"type":"response","@timestamp":"2019-01-07T12:57:47Z","tags":[],"pid":2886,"method":"get","statusCode":500,"req":{"url":"/app/kibana","method":"get","headers":{"connection":"upgrade","host":"main_node","cache-
Jan 07 12:57:47 main_node kibana[2886]: {"type":"log","@timestamp":"2019-01-07T12:57:47Z","tags":["info","authentication"],"pid":2886,"message":"Authentication attempt failed: [exception] Security must be explicitly enabled when using a trial license. Enable security
Jan 07 12:57:47 main_node kibana[2886]: {"type":"error","@timestamp":"2019-01-07T12:57:47Z","tags":[],"pid":2886,"level":"error","error":{"message":"[exception] Security must be explicitly enabled when using a trial license. Enable security by setting [xpack.security
Jan 07 12:57:47 main_node kibana[2886]: exception] Security must be explicitly enabled when using a trial license. Enable security by setting [xpack.security.enabled] to [true] in the elasticsearch.yml file and restart the node."}
```
So I went in the main node YML file and added the following:
```
xpack.security.enabled: true
```
As suggested in the official documentation: [Enable Security](https://www.elastic.co/guide/en/elastic-stack-overview/current/get-started-enable-security.html) but as far as I was aware, this would be a paid feature with Xpack which I am not yet looking to get into as I am still testing and learning.
When I add the line, this is the error I see:
```
root@main_node:~# sudo service kibana status
● kibana.service - Kibana
Loaded: loaded (/etc/systemd/system/kibana.service; enabled; vendor preset: enabled)
Active: active (running) since Mon 2019-01-07 13:00:19 GMT; 2min 0s ago
Main PID: 3205 (node)
Tasks: 10
Memory: 272.0M
CPU: 5.740s
CGroup: /system.slice/kibana.service
└─3205 /usr/share/kibana/bin/../node/bin/node --no-warnings /usr/share/kibana/bin/../src/cli -c /etc/kibana/kibana.yml
Jan 07 13:01:03 main_node kibana[3205]: {"type":"log","@timestamp":"2019-01-07T13:01:03Z","tags":["status","plugin:rollup@6.5.2","error"],"pid":3205,"state":"red","message":"Status changed from red to red - [security_exception] missing authentication token for REST r
Jan 07 13:01:03 main_node kibana[3205]: {"type":"log","@timestamp":"2019-01-07T13:01:03Z","tags":["status","plugin:graph@6.5.2","error"],"pid":3205,"state":"red","message":"Status changed from red to red - [security_exception] missing authentication token for REST re
Jan 07 13:01:03 main_node kibana[3205]: {"type":"log","@timestamp":"2019-01-07T13:01:03Z","tags":["status","plugin:spaces@6.5.2","error"],"pid":3205,"state":"red","message":"Status changed from red to red - [security_exception] missing authentication token for REST r
Jan 07 13:01:03 main_node kibana[3205]: {"type":"log","@timestamp":"2019-01-07T13:01:03Z","tags":["status","plugin:security@6.5.2","error"],"pid":3205,"state":"red","message":"Status changed from red to red - [security_exception] missing authentication token for REST
Jan 07 13:01:03 main_node kibana[3205]: {"type":"log","@timestamp":"2019-01-07T13:01:03Z","tags":["status","plugin:grokdebugger@6.5.2","error"],"pid":3205,"state":"red","message":"Status changed from red to red - [security_exception] missing authentication token for
Jan 07 13:01:03 main_node kibana[3205]: {"type":"log","@timestamp":"2019-01-07T13:01:03Z","tags":["status","plugin:logstash@6.5.2","error"],"pid":3205,"state":"red","message":"Status changed from red to red - [security_exception] missing authentication token for REST
Jan 07 13:01:03 main_node kibana[3205]: {"type":"log","@timestamp":"2019-01-07T13:01:03Z","tags":["status","plugin:beats_management@6.5.2","error"],"pid":3205,"state":"red","message":"Status changed from red to red - [security_exception] missing authentication token
Jan 07 13:01:03 main_node kibana[3205]: {"type":"log","@timestamp":"2019-01-07T13:01:03Z","tags":["status","plugin:reporting@6.5.2","error"],"pid":3205,"state":"red","message":"Status changed from red to red - [security_exception] missing authentication token for RES
Jan 07 13:01:25 main_node kibana[3205]: {"type":"log","@timestamp":"2019-01-07T13:01:25Z","tags":["license","warning","xpack"],"pid":3205,"message":"License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. [security_
Jan 07 13:01:55 main_node kibana[3205]: {"type":"log","@timestamp":"2019-01-07T13:01:55Z","tags":["license","warning","xpack"],"pid":3205,"message":"License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. [security_
```
Elasticsearch appears to be running with no issues in the logs on both nodes but I could be wrong.
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [January 7, 2019, 1:37pm UTC](https://discuss.elastic.co/t/cant-see-the-second-node-whilst-attempting-to-build-a-cluster/163187/4 "2019-01-07T13:37:10Z")
Ok, if the Elasticsearch logs seem to be clear then I'm not the best person to help. Can you try `GET _cluster/health` from the command line (i.e. something like `curl http://192.168.0.64:9200/_cluster/health`) to confirm that Elasticsearch really is running and has successfully formed a cluster of 2 nodes? If it has, it's probably best to ask about these Kibana messages in the [Kibana forum](https://discuss.elastic.co/c/kibana).
---
### Author:  [@sc1](https://discuss.elastic.co/u/sc1)
#### Post date: [January 7, 2019, 4:04pm UTC](https://discuss.elastic.co/t/cant-see-the-second-node-whilst-attempting-to-build-a-cluster/163187/5 "2019-01-07T16:04:49Z")
Hey @DavidTurner
The only changes I have done are the Discovery Zen section, is there somewhere else I am supposed to change ? For example, the documentation mentions changing nodes to Master/Data nodes/Ingest nodes but I can't see any of that in my default yml file?
I changed the discovery\_zen.minimum\_master\_nodes back to 1 and my main node works again and doesn't throw any of the above errors, it's only when I change it to 2 it shows errors. So I think the issue is still in just Elasticsearch.
---
### Author:  [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)
#### Post date: [January 7, 2019, 4:21pm UTC](https://discuss.elastic.co/t/cant-see-the-second-node-whilst-attempting-to-build-a-cluster/163187/6 "2019-01-07T16:21:51Z")
The errors above are from Kibana's log, which is why I think you should ask about them in the Kibana forum. If you think Elasticsearch is not working right now, it'd be best to take Kibana out of the equation for now and just focus on the Elasticsearch part.
> [@sc1](#):
>
> I changed the discovery\_zen.minimum\_master\_nodes back to 1
It absolutely must be `2` if you have 2 master-eligible nodes in your cluster.
---
### Author:  [@sc1](https://discuss.elastic.co/u/sc1)
#### Post date: [January 16, 2019, 11:10am UTC](https://discuss.elastic.co/t/cant-see-the-second-node-whilst-attempting-to-build-a-cluster/163187/7 "2019-01-16T11:10:07Z")
I have asked the following question on the Kibana forum. As I am still facing the issue, any help would be greatly appreciated.
[Question in Kibana Forum](https://discuss.elastic.co/t/adding-a-second-to-cluster-kibana-errors/164129)
---
### Author:  [@tsullivan](https://discuss.elastic.co/u/tsullivan)
#### Post date: [January 24, 2019, 10:28pm UTC](https://discuss.elastic.co/t/cant-see-the-second-node-whilst-attempting-to-build-a-cluster/163187/8 "2019-01-24T22:28:40Z")
Hi, the messages you were seeing that you had to explicitly enable Security might have been showing up because you had `xpack.security` settings in your `elasticsearch.yml` file. ES warned you that it can't do anything with those settings because Security wasn't automatically enabled for your Trial license.
If that was the case, you don't need to enable security, just take out the settings from your `elasticsearch.yml` that try to configure security features.
From there on, it looks like Kibana was failing to start up completely because it could not authenticate with Elasticsearch. That could have happened because no users were created in ES, or `kibana.yml` did not have the auth config lines (`elasticsearch.username`, `elasticsearch.password`) or it had the incorrect auth. Since it says `missing authentication token` I think that means you had no auth settings in your `kibana.yml`
Other than `node.master: true` in one node and `node.data: true` in another, is everything in the ES configs exactly the same? Since security is popping up as a trouble spot, you might have had different security settings in one of the configs.
> Networking wise, the two hosts are able to communicate and both configs are near enough identical
How do you determine the network status? To verify what you say, you'd need to check:
```
curl https://elasticsearch:9200/_cluster/health
```
and look for:
```
"number_of_nodes": 2
```
in the response. Invalid security settings will certainly prevent a node from joining a cluster, so check for that.
Hope that helps!
---
### Author:  [@sc1](https://discuss.elastic.co/u/sc1)
#### Post date: [January 25, 2019, 2:42pm UTC](https://discuss.elastic.co/t/cant-see-the-second-node-whilst-attempting-to-build-a-cluster/163187/9 "2019-01-25T14:42:42Z")
Good afternoon Tim, thank you for the response.
I have gone through what you've said and can confirm that `xpack.security` settings are all off on all 3 `elasticsearch.yml` files. As I am using the same version 6.5.4 on all 3 servers, the .yml is identical on all 3.
In regards to verifying the network, I had been able to telnet to all hosts and that showed that the network was working. I have however used the method you suggested and here are the results:
The following results are from the two nodes I am trying to add, where I seem to have no issues with Elasticsearch running.
```
{"cluster_name":"adfs-logs","status":"green","timed_out":false,"number_of_nodes":2,"number_of_data_nodes":2,"active_primary_shards":0,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":100.0}
```
And from the second node:
```
{"cluster_name":"adfs-logs","status":"green","timed_out":false,"number_of_nodes":2,"number_of_data_nodes":2,"active_primary_shards":0,"active_shards":0,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":0,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":100.0}[root@lcs-admqdjg-pc elasticsearch]
```
When I run the command on the main server:
```
{"error":{"root_cause":[{"type":"master_not_discovered_exception","reason":null}],"type":"master_not_discovered_exception","reason":null},"status":503}
```
Kibana is still throwing the license errors, but literally, when what's frustrating is it seems to want to function just as a single node. Let me know if there's more logs you require.
---
### Author:  [@tsullivan](https://discuss.elastic.co/u/tsullivan)
#### Post date: [January 25, 2019, 11:21pm UTC](https://discuss.elastic.co/t/cant-see-the-second-node-whilst-attempting-to-build-a-cluster/163187/10 "2019-01-25T23:21:40Z")
Hi, this info is great! I formatted the JSON with `jq` and started looking.
When you have 2 nodes joined in a cluster, they both report:
> "number\_of\_nodes": 2,
> "number\_of\_data\_nodes": 2,
Something is wrong here. There always needs to be an elected master in the cluster so each data node can be notified of updates to the cluster state. You have both nodes as data nodes, instead of either being elected as master.
Both nodes also show:
> "active\_primary\_shards": 0,
> "active\_shards": 0,
> "relocating\_shards": 0,
> "initializing\_shards": 0,
> "unassigned\_shards": 0,
The problem that's stopping Kibana from functioning doesn't look like a License issue. Without an elected master node, you will not be able to create an index and without any shards, there is nothing to query. I think the 3rd part you captured is to that effect. This: `master_not_discovered_exception` needs to be resolved.
I'd like you to take a look more at the documentation on the ES node types, especially master node: [Node | Elasticsearch Guide [8.11] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#master-node)
If that's not helping, let's get as much of the `elasticsearch.yml` contents as you can post here (no confidential info please).
We'd have to look for any settings that are preventing the nodes from being master-eligible.
You have nothing like this in your elasticsearch.yml, right?
```
node.master:false
```
Or could there be any possible firewall issues? The nodes need to be able to talk to each other in TCP port 9300.
---
### Author:  [@tsullivan](https://discuss.elastic.co/u/tsullivan)
#### Post date: [January 25, 2019, 11:24pm UTC](https://discuss.elastic.co/t/cant-see-the-second-node-whilst-attempting-to-build-a-cluster/163187/11 "2019-01-25T23:24:01Z")
BTW what does `/_cluster/health` show when you have just a single node?
When you say it is working with the 1 node, does that mean you are able to index data and search on it (either in curl or in Kibana)?
---
### Author:  [@sc1](https://discuss.elastic.co/u/sc1)
#### Post date: [January 28, 2019, 2:34pm UTC](https://discuss.elastic.co/t/cant-see-the-second-node-whilst-attempting-to-build-a-cluster/163187/12 "2019-01-28T14:34:00Z")
Hi Tim, apologies for the delayed response.
When running as a single node, I face 0 issues. I am able to access via Kibana, process logs, build dashboards etc.
Here is the `elasticsearch.yml` as requested:
```
# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: adfs-logs
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: "test-node-1"
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: localhost
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when new node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.zen.ping.unicast.hosts: ["XXX.XXX.XXX.64", "XXX.XXX.XXX..63:9300", "XXX.XXX.XXX.139:9300"]
#
# Prevent the "split brain" by configuring the majority of nodes (total number of master-eligible nodes / 2 + 1):
#
discovery.zen.minimum_master_nodes: 3
#
# For more information, consult the zen discovery module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true
xpack.security.enabled: false
```
And the curl request you asked for when it's working:
```
"cluster_name":"adfs-logs","status":"yellow","timed_out":false,"number_of_nodes":1,"number_of_data_nodes":1,"active_primary_shards":716,"active_shards":716,"relocating_shards":0,"initializing_shards":0,"unassigned_shards":703,"delayed_unassigned_shards":0,"number_of_pending_tasks":0,"number_of_in_flight_fetch":0,"task_max_waiting_in_queue_millis":0,"active_shards_percent_as_number":50.45806906272022}
```
Let me know if there's anything else you need.
---
### Author:  [@tsullivan](https://discuss.elastic.co/u/tsullivan)
#### Post date: [January 31, 2019, 12:42am UTC](https://discuss.elastic.co/t/cant-see-the-second-node-whilst-attempting-to-build-a-cluster/163187/13 "2019-01-31T00:42:03Z")
In your elasticsearch.yml, you have:
> network.host: localhost
That is making Elasticsearch listen just on the loopback interface: it'll only be able to connect to requests that were made to `localhost`. The nodes will need to bind to a non-loopback address. See [https://www.elastic.co/guide/en/elasticsearch/reference/current/network.host.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/network.host.html)
---
### Author:  [@sc1](https://discuss.elastic.co/u/sc1)
#### Post date: [February 1, 2019, 12:44pm UTC](https://discuss.elastic.co/t/cant-see-the-second-node-whilst-attempting-to-build-a-cluster/163187/14 "2019-02-01T12:44:07Z")
Good afternoon Tim,
I followed said advice and removed `network.host: localhost` on all the `yml` files and this has resolved the problem. The cluster is now live!!!! Can't thank you enough.
---
### Author:  [@tsullivan](https://discuss.elastic.co/u/tsullivan)
#### Post date: [February 1, 2019, 4:30pm UTC](https://discuss.elastic.co/t/cant-see-the-second-node-whilst-attempting-to-build-a-cluster/163187/15 "2019-02-01T16:30:24Z")
Fantastic! Sorry it took so long to get there, but glad it worked out!
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 1, 2019, 4:30pm UTC](https://discuss.elastic.co/t/cant-see-the-second-node-whilst-attempting-to-build-a-cluster/163187/16 "2019-03-01T16:30:28Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.