# ERROR: Failed to determine the health of the cluster
**URL:** https://discuss.elastic.co/t/error-failed-to-determine-the-health-of-the-cluster/328746
**Category:** Elasticsearch
**Created:** [March 28, 2023, 6:48pm UTC](https://discuss.elastic.co/t/error-failed-to-determine-the-health-of-the-cluster/328746 "2023-03-28T18:48:30Z")
**Posts on this page:** 13
**Page:** 1
### Author:  [@DRW-ATCA](https://discuss.elastic.co/u/DRW-ATCA)
#### Post date: [March 28, 2023, 6:48pm UTC](https://discuss.elastic.co/t/error-failed-to-determine-the-health-of-the-cluster/328746/1 "2023-03-28T18:48:30Z")
I am trying to add nodes to a cluster for ES 8.6.2 in an AWS EC2 environment. After creating a master node and the first of several data nodes, the two instances give healthy responses to the following commands but do not recognize each other:
systemctl status elasticsearch
curl -X GET [http://127.0.0.1:9200/\_cluster/health?pretty](http://127.0.0.1:9200/_cluster/health?pretty)
curl -X GET [http://10.0.140.82:9200/\_cluster/health?pretty](http://10.0.140.82:9200/_cluster/health?pretty) (\<\< private VPC IP)
To each of the following commands in the master node, the response is always
"ERROR: Failed to determine the health of the cluster":
/usr/share/elasticsearch/bin/elasticsearch-create-enrollment-token -s node
elasticsearch-create-enrollment-token -s node
elasticsearch-create-enrollment-token -s node
elasticsearch-reset-password
Same results using 'sudo' and '-f'.
How can I create an enrollment token?
Thanks
---
### Author:  [@warkolm](https://discuss.elastic.co/u/warkolm)
#### Post date: [March 28, 2023, 10:44pm UTC](https://discuss.elastic.co/t/error-failed-to-determine-the-health-of-the-cluster/328746/2 "2023-03-28T22:44:05Z")
Welcome to our community! 😃
Please share more logs, and your Elasticsearch configs.
---
### Author:  [@DRW-ATCA](https://discuss.elastic.co/u/DRW-ATCA)
#### Post date: [March 28, 2023, 11:13pm UTC](https://discuss.elastic.co/t/error-failed-to-determine-the-health-of-the-cluster/328746/3 "2023-03-28T23:13:01Z")
````auto
======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: usw2
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: master
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: 127.0.0.1
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# --------------------------------- Readiness ----------------------------------
#
# Enable an unauthenticated TCP readiness endpoint on localhost
#
#readiness.port: 9399
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
#action.destructive_requires_name: false
#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically
# generated to configure Elasticsearch security features on 28-03-2023 00:51:05
#
# --------------------------------------------------------------------------------
# Enable security features
xpack.security.enabled: false
xpack.security.enrollment.enabled: true
# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
enabled: true
keystore.path: certs/http.p12
# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
keystore.path: certs/transport.p12
truststore.path: certs/transport.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
cluster.initial_master_nodes: ["ip-10-0-140-82.us-west-2.compute.internal", "ip-10-0-136-138.us-west-2.compute.internal"]
# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0
# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
#transport.host: 0.0.0.0
#----------------------- END SECURITY AUTO CONFIGURATION -------------------------```
````
---
### Author:  [@DRW-ATCA](https://discuss.elastic.co/u/DRW-ATCA)
#### Post date: [March 28, 2023, 11:20pm UTC](https://discuss.elastic.co/t/error-failed-to-determine-the-health-of-the-cluster/328746/4 "2023-03-28T23:20:12Z")
````auto
======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
# Before you set out to tweak and tune the configuration, make sure you
# understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: usw2
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: data1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/lib/elasticsearch
#
# Path to log files:
#
path.logs: /var/log/elasticsearch
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: 127.0.0.1
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
discovery.seed_hosts:
- 10.0.140.82
# - 10.0.140.82:9200
# - ip-10-0-140-82.us-west-2.compute.internal
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#cluster.initial_master_nodes:
# - es-usw2-master
# - 10.0.140.82
#
# For more information, consult the discovery and cluster formation module documentation.
#
# --------------------------------- Readiness ----------------------------------
#
# Enable an unauthenticated TCP readiness endpoint on localhost
#
#readiness.port: 9399
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
#action.destructive_requires_name: false
#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically
# generated to configure Elasticsearch security features on 23-03-2023 20:03:00
#
# --------------------------------------------------------------------------------
# Enable security features
xpack.security.enabled: false
xpack.security.enrollment.enabled: true
# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
enabled: true
keystore.path: certs/http.p12
# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
keystore.path: certs/transport.p12
truststore.path: certs/transport.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
#cluster.initial_master_nodes: ["ip-10-0-140-82.us-west-2.compute.internal"]
cluster.initial_master_nodes: ["10.0.140.82"]
# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0
# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
#transport.host: 0.0.0.0
#----------------------- END SECURITY AUTO CONFIGURATION -------------------------```
````
---
### Author:  [@DRW-ATCA](https://discuss.elastic.co/u/DRW-ATCA)
#### Post date: [March 28, 2023, 11:24pm UTC](https://discuss.elastic.co/t/error-failed-to-determine-the-health-of-the-cluster/328746/5 "2023-03-28T23:24:15Z")
Please find the config files of the master and data node posted above, and curl results below, in the order mentioned.
```auto
sh-4.2$ curl -X GET http://10.0.140.82:9200/_cluster/health?pretty
{
"cluster_name" : "usw2",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 1,
"active_shards" : 1,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
```
````auto
sh-4.2$ curl -X GET http://10.0.140.82:9200/_cluster/health?pretty
{
"cluster_name" : "usw2",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 1,
"number_of_data_nodes" : 1,
"active_primary_shards" : 1,
"active_shards" : 1,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 0,
"delayed_unassigned_shards" : 0,
"number_of_pending_tasks" : 0,
"number_of_in_flight_fetch" : 0,
"task_max_waiting_in_queue_millis" : 0,
"active_shards_percent_as_number" : 100.0
}
sh-4.2$```
````
---
### Author:  [@rachel\_gomez](https://discuss.elastic.co/u/rachel_gomez)
#### Post date: [March 29, 2023, 5:08am UTC](https://discuss.elastic.co/t/error-failed-to-determine-the-health-of-the-cluster/328746/6 "2023-03-29T05:08:20Z")
I encountered the same problem, and I just redid the process - unzipped the ES and kibana zip files again, and ran bin/elasticsearch in the newly created directory. Look for a message that is encapsulated in a formatted box that contains both the password for the elastic user, and the enrollment token for Kibana (the token is only valid for 30 minutes). This message will only appear once, the first time you run elasticsearch.
I proceeded to run bin/kibana for Kibana and configured it in the browser, and everything worked out from there. Hope this helps!
Regards,
Rachel Gomez
---
### Author:  [@DRW-ATCA](https://discuss.elastic.co/u/DRW-ATCA)
#### Post date: [March 29, 2023, 12:21pm UTC](https://discuss.elastic.co/t/error-failed-to-determine-the-health-of-the-cluster/328746/7 "2023-03-29T12:21:12Z")
Hi, thank you for your comment. Sadly, my experience differs from yours. I get the formatted box as you describe but the enrollment process fails with errors when trying to use the tokens.
---
### Author:  [@DRW-ATCA](https://discuss.elastic.co/u/DRW-ATCA)
#### Post date: [March 29, 2023, 8:16pm UTC](https://discuss.elastic.co/t/error-failed-to-determine-the-health-of-the-cluster/328746/8 "2023-03-29T20:16:00Z")
attempted a different solution
ref: [Joining instances to form cluster](https://discuss.elastic.co/t/joining-instances-to-form-cluster/328860)
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [March 29, 2023, 8:24pm UTC](https://discuss.elastic.co/t/error-failed-to-determine-the-health-of-the-cluster/328746/9 "2023-03-29T20:24:13Z")
Unless all nodes are running on the same host you need to set netwok.host to 0.0.0.0 or the ip address of the host.
---
### Author:  [@DRW-ATCA](https://discuss.elastic.co/u/DRW-ATCA)
#### Post date: [March 29, 2023, 9:19pm UTC](https://discuss.elastic.co/t/error-failed-to-determine-the-health-of-the-cluster/328746/10 "2023-03-29T21:19:46Z")
Thank you for the reply. We previously tried the host IP, no change.
With 0.0.0.0,
`"type " : "master_not_discovered_exception ", "reason " : null `
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [March 30, 2023, 5:44am UTC](https://discuss.elastic.co/t/error-failed-to-determine-the-health-of-the-cluster/328746/11 "2023-03-30T05:44:13Z")
I would recommend you have a look [at the docs](https://www.elastic.co/guide/en/elasticsearch/reference/8.6/modules-discovery-bootstrap-cluster.html). In order for Elasticsearch nodes to be able to communicate and form a cluster `network.host` must be set to an IP address the other nodes can reach, e.g. `0.0.0.0` (all interfaces).
```auto
node.name: master
```
```auto
cluster.initial_master_nodes: ["ip-10-0-140-82.us-west-2.compute.internal", "ip-10-0-136-138.us-west-2.compute.internal"]
```
As outlined in the docs I linked to:
> The node names used in the `cluster.initial_master_nodes` list must exactly match the `node.name` properties of the nodes.
This is something you need to fix. There may be other issues as well, so please go through the docs carefully.
I would also recommend having 3 master eligible nodes in the cluster [as fewer than this will mean the cluster is not highly available](https://www.elastic.co/guide/en/elasticsearch/reference/8.6/high-availability-cluster-small-clusters.html).
---
### Author:  [@DRW-ATCA](https://discuss.elastic.co/u/DRW-ATCA)
#### Post date: [March 30, 2023, 5:20pm UTC](https://discuss.elastic.co/t/error-failed-to-determine-the-health-of-the-cluster/328746/12 "2023-03-30T17:20:52Z")
Thank you for your comments. Making the changes you noted and deleting some files on the data1 node solved the issue.
Misalignment of definitions in the yml files caused the nodes to form independent clusters.
Deleting files specified in the data1 logs (below), aligning definitions and restarting solved the issue.
> "Note that the cluster UUID persists across restarts and can only be changed by deleting the contents of the node's data path [/var/lib/elasticsearch] which will also remove any data held by this node."
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 27, 2023, 5:21pm UTC](https://discuss.elastic.co/t/error-failed-to-determine-the-health-of-the-cluster/328746/13 "2023-04-27T17:21:24Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.