Need help setting up a second node

Elastic Stack Elasticsearch
1

Hello!

I have a server with Elasticsearch, Kibana and a fleet-server agent installed.
I use all the default certificates created in the beginning (automatic security configuration I believe it's called?).

I would like to add a second server with Elasticsearch (a second node). I've tried using the enrollment token but it says it can't reach the node:

Unable to communicate with the node on ://127.0.0.1:9200/_security/enroll/node. Error was Connection refused
ERROR: Aborting enrolling to cluster. Could not communicate with the node on any of the addresses from the enrollment token. All of [127.0.0.1:9200] were attempted., with exit code 69

What do I need to do for this enrollment token to work? Is there another way to enroll the node?

Every help and piece of information is welcome!

Thanks!

2

Hi @hairless_mess Welcome to the community.

We will need some basics to help

What documentation did you follow?

What Version?

How did you install elasticsearch?

Please show the full command(s) you used for the enrollment process (creating the token, applying the token)

Share the elasticsearch.yml for both nodes?

Perhaps with that we can help.

3

Hello!

I followed this documentation:

I'm using version 8.17.0

I deployed it using the .tar package

On the master node:
elasticsearch-create-enrollment-token -s node

On the new node:
elasticsearch --enrollment-token <enrollment-token>

This is the config for my master node:

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: my-machine
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: master-node
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/opt/elasticsearch/data
#
# Path to log files:
#
path.logs: /var/opt/elasticsearch/log
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
#network.host: 0.0.0.0
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: [""]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
#action.destructive_requires_name: false

#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically
# generated to configure Elasticsearch security features on 21-06-2024 08:07:18
#
# --------------------------------------------------------------------------------

# Enable security features
xpack.security.enabled: true

xpack.security.enrollment.enabled: true

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
cluster.initial_master_nodes: ["my-machine"]

# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0

# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
#transport.host: 0.0.0.0

#----------------------- END SECURITY AUTO CONFIGURATION -------------------------

This is for the second node:

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
#
# Path to log files:
#
#path.logs: /path/to/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
#network.host: 192.168.0.1
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
#action.destructive_requires_name: false
4

Notice at the bottom of the first .yml

Uncomment the transport.host: 0.0.0.0

# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
#transport.host: 0.0.0.0

I would review this section

Additionally, only nodes on the same host can join the cluster without additional configuration. If you want nodes from another host to join your cluster, you need to set transport.host to a supported value (such as uncommenting the suggested value of 0.0.0.0), or an IP address that’s bound to an interface where other hosts can reach it. Refer to transport settings for more information.

5

After uncommenting the line, restarting elasticsearch, recreating the enrollment, and trying again to enroll the node, I get the same error.

6

Please share the exact command you ran with the exact error message.

7

./elasticsearch --enrollment-token XXXXXX==

Unable to communicate with the node on https://127.0.0.1:9200/_security/enroll/node. Error was Connection refused
ERROR: Aborting enrolling to cluster. Could not communicate with the node on any of the addresses from the enrollment token. All of [127.0.0.1:9200] were attempted., with exit code 69
8

Hmmmm

On the initial node Can you try putting the IP address of that machine in and try the restart regenerate etc...

http.host: 192.168.10.1
transport.host:  192.168.10.1

Also as a check from the second machine

curl -v -k -u elastic https://<ip_of_initial>:9200

9

I think it worked, how can I check if the node joined the cluster?

I had to change some little things here and there, I will post everything when I'm done.

10

Either of these

GET  _cat/nodes?v
GET _cluster/health
11

Not working yet. When I run the enroll command I get this at the end:

[2025-03-19T17:37:32,075][INFO ][o.e.c.c.ClusterBootstrapService] [second-node] this node has not joined a bootstrapped cluster yet; [cluster.initial_master_nodes] is set to []
[2025-03-19T17:37:32,925][INFO ][o.e.c.c.JoinHelper       ] [second-node] failed to join {master-node}{QMruOrUFSZS_wCuvsnT8BQ}{89tCkQwiQziGxsFOq_HuCg}{master-node}{master-node-IP}{[master-node-IP]:9300}{cdfhilmrstw}{8.17.0}{7000099-8521000}{ml.max_jvm_size=4148166656, ml.config_version=12.0.0, xpack.installed=true, transform.config_version=10.0.0, ml.machine_memory=8290643968, ml.allocated_processors=2, ml.allocated_processors_double=2.0} with JoinRequest{sourceNode={second-node}{umreEYrATDGOhfWw6FJ2tQ}{xdfoDXgmRuyEtd71dyl_Qw}{second-node}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw}{8.17.0}{7000099-8521000}{ml.max_jvm_size=4148166656, ml.config_version=12.0.0, xpack.installed=true, transform.config_version=10.0.0, ml.machine_memory=8290635776, ml.allocated_processors=2, ml.allocated_processors_double=2.0}, compatibilityVersions=CompatibilityVersions[transportVersion=8797002, systemIndexMappingsVersion={.secrets-inference=MappingsVersion[version=1, hash=-1434574148], .triggered_watches=MappingsVersion[version=1, hash=-502826165], .fleet-agents-7=MappingsVersion[version=1, hash=-270798539], .fleet-servers-7=MappingsVersion[version=1, hash=-916922632], .fleet-policies-leader-7=MappingsVersion[version=1, hash=-1108172796], .ml-config=MappingsVersion[version=1, hash=-319778629], .security-tokens-7=MappingsVersion[version=1, hash=576296021], .geoip_databases=MappingsVersion[version=1, hash=-305757839], .snapshot-blob-cache=MappingsVersion[version=1, hash=632712485], .security-profile-8=MappingsVersion[version=2, hash=-909540896], .search-app-1=MappingsVersion[version=1, hash=-501711141], .watches=MappingsVersion[version=1, hash=-1045118511], .fleet-artifacts-7=MappingsVersion[version=1, hash=-1593703898], .query-rules-2=MappingsVersion[version=2, hash=-1272560824], .transform-internal-007=MappingsVersion[version=1, hash=1144737897], .fleet-enrollment-api-keys-7=MappingsVersion[version=1, hash=-1804942283], .fleet-actions-7=MappingsVersion[version=1, hash=-2624357], .tasks=MappingsVersion[version=0, hash=-945584329], .fleet-secrets-7=MappingsVersion[version=1, hash=-745394230], .ml-meta=MappingsVersion[version=2, hash=-613742866], .security-7=MappingsVersion[version=3, hash=-832976091], .connector-secrets-1=MappingsVersion[version=1, hash=-745394230], .logstash=MappingsVersion[version=1, hash=-1058806351], .ml-inference-000005=MappingsVersion[version=3, hash=919553140], .async-search=MappingsVersion[version=0, hash=-1403744380], .inference=MappingsVersion[version=2, hash=-1459421596], .fleet-policies-7=MappingsVersion[version=1, hash=-201702522], .synonyms-2=MappingsVersion[version=1, hash=-888080772]}], features=[simulate.mapping.addition, mapper.vectors.bbq, text_similarity_reranker_retriever_composition_supported, esql.casting_operator, standard_retriever_supported, flattened.ignore_above_support, features_supported, stats.include_disk_thresholds, mapper.ignore_above_index_level_setting, data_stream.rollover.lazy, routing.multi_value_routing_path, get_database_configuration_action.multi_node, desired_node.version_deprecated, search.vectors.k_param_supported, mapper.source.synthetic_source_with_copy_to_and_doc_values_false, mapper.source.synthetic_source_fallback, mapper.range.null_values_off_by_one_fix, mapper.keyword_normalizer_synthetic_source, mapper.subobjects_auto, mapper.keyword_dimension_ignore_above, script.term_stats, esql.mv_sort, esql.mv_ordering_sorted_ascending, health.dsl.info, esql.spatial_points_from_source, mapper.query_index_mode, esql.st_x_y, esql.base64_decode_encode, routing.boolean_routing_path, mapper.pass_through_priority, esql.async_query, unified_highlighter_matched_fields, esql.agg_values, geoip.downloader.database.configuration, mapper.track_ignored_source, retrievers_supported, put_database_configuration_action.ipinfo, rest.local_only_capabilities, text_similarity_reranker_retriever_supported, simulate.component.template.substitutions, esql.st_centroid_agg, query_rules.test, mapper.index_sorting_on_nested, security.roles_metadata_flattened, license-trial-independent-version, esql.disable_nullable_opts, script.hamming, rest.capabilities_action, mapper.source.synthetic_source_stored_fields_advance_fix, esql.resolve_fields_api, mapper.flattened.ignore_above_with_arrays_support, health.extended_repository_indicator, mapper.segment_level_fields_stats, esql.counter_types, knn_retriever_supported, mapper.subobjects_auto_fixes, mapper.vectors.bit_vectors, esql.spatial_shapes, data_stream.auto_sharding, esql.metrics_counter_fields, data_stream.lifecycle.global_retention, mapper.boolean_dimension, semantic_text.default_elser_2, logsdb_telemetry, usage.data_tiers.precalculate_stats, esql.st_intersects, random_reranker_retriever_supported, cluster.stats.source_modes, simulate.mapping.validation.templates, esql.st_contains_within, snapshot.repository_verify_integrity, esql.st_disjoint, rrf_retriever_composition_supported, mapper.source.synthetic_source_copy_to_fix, query_rule_retriever_supported, esql.timespan_abbreviations, mapper.synthetic_source_keep, query_rule_list_types, mapper.source.synthetic_source_copy_to_inside_objects_fix, tsdb.ts_routing_hash_doc_value_parse_byte_ref, logsdb_telemetry_stats, simulate.index.template.substitutions, file_settings, esql.string_literal_auto_casting, repositories.supports_usage_stats, rrf_retriever_supported, esql.from_options, mapper.vectors.int4_quantization, semantic_text.search_inference_id, esql.string_literal_auto_casting_extended, slm.interval_schedule, security.role_mapping_cleanup, security.migration_framework, simulate.mapping.validation, simulate.support.non.template.mapping, esql.metadata_fields], minimumTerm=37, optionalJoin=Optional[Join[votingNode={second-node}{umreEYrATDGOhfWw6FJ2tQ}{xdfoDXgmRuyEtd71dyl_Qw}{second-node}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw}{8.17.0}{7000099-8521000}{ml.max_jvm_size=4148166656, ml.config_version=12.0.0, xpack.installed=true, transform.config_version=10.0.0, ml.machine_memory=8290635776, ml.allocated_processors=2, ml.allocated_processors_double=2.0}, masterCandidateNode={master-node}{QMruOrUFSZS_wCuvsnT8BQ}{89tCkQwiQziGxsFOq_HuCg}{master-node}{master-node-IP}{[master-node-IP]:9300}{cdfhilmrstw}{8.17.0}{7000099-8521000}{ml.max_jvm_size=4148166656, ml.config_version=12.0.0, xpack.installed=true, transform.config_version=10.0.0, ml.machine_memory=8290643968, ml.allocated_processors=2, ml.allocated_processors_double=2.0}, term=37, lastAcceptedTerm=0, lastAcceptedVersion=0]]}org.elasticsearch.transport.RemoteTransportException: [master-node][[master-node-ip]:9300][internal:cluster/coordination/join]
12

can you share the current state of the second nodes .yml

Unfortunately I think you are "half setup"

If you set this in the 2nd .yml

cluster.initial_master_nodes: <ip_of_first_node>

Start it I suspect the cluster will form.. but then you need to take that setting out... it is a 1 time setting see here

Then just try to resart the 2nd node without the enrollment token etc... i.e. just regular and see if you have the 2 nodes

13

This is my file after running the enroll command, without adding the cluster.initial_master_nodes:

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: my-cluster
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
#
# Path to log files:
#
#path.logs: /path/to/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
#network.host: 192.168.0.1
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["[myIP]"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
#action.destructive_requires_name: false

#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically
# generated to configure Elasticsearch security features on 20-03-2025 10:34:40
#
# --------------------------------------------------------------------------------

# Enable security features
xpack.security.enabled: true

xpack.security.enrollment.enabled: true

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
# Discover existing nodes in the cluster
discovery.seed_hosts: ["[myIP]:9300"]

# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0

# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
transport.host: 0.0.0.0

#----------------------- END SECURITY AUTO CONFIGURATION -------------------------

I tried doing like you said by adding the cluster.initial_master_nodes and restarting and then removing it and restarting again but it didn't work.

14

1st node you have

2nd Node you have

Those will not form a single cluster AFAIK as the cluster names are different.

What does that mean it did not work, please always include the diescription and error..

It did not work... the node did not start?

It did not work ... the node started but was not part of the cluster?

Change the cluster name in the 2nd put the master nodes in...

Start the cluster see if you see the 2 nodes... if so then remove the initial master node.

You may need to clean up the 2nd node (including data) and try to re-enroll

15

I have the same cluster name in both nodes, it was an error from my part when copying the configs here, my bad.

The node starts but it's not in the cluster.

I have done the steps you mention but the node still doesn't join the cluster.

I always restart from a clean snapshot of the machine before the enrollment.

16

Hmmm I am not sure what is going on...

You are following these instructions precisely? I have done this many times...

Including cleaning up the data directory. If you have not done that, this could explain a lot.

Are you still working with the initial node #1 or did you start over with that?

Can you share the logs where the cluster is forming from the 2nd node. This is important...

Can you share the current state of the 2 elasticsearch.yml.

17

The node #1 hasn't changed.

I deleted the contents of the data folder and ran the enrollment command again with the same result:

...
[2025-03-25T10:13:11,632][INFO ][o.e.t.TransportService   ] [lab-elk-1] publish_address {127.0.0.1:9300}, bound_addresses {[::]:9300}
[2025-03-25T10:13:11,772][INFO ][o.e.b.BootstrapChecks    ] [lab-elk-1] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2025-03-25T10:13:11,776][INFO ][o.e.c.c.ClusterBootstrapService] [lab-elk-1] this node has not joined a bootstrapped cluster yet; [cluster.initial_master_nodes] is set to []
[2025-03-25T10:13:12,774][INFO ][o.e.c.c.JoinHelper       ] [lab-elk-1] failed to join {master-node}{QMruOrUFSZS_wCuvsnT8BQ}{89tCkQwiQziGxsFOq_HuCg}
...

Whenever I run the enrollment command it keeps sending this error message until I do a CTRL+C to stop it. I then added the initial master node in the config and tried again. This gave me the same result:

...
[2025-03-25T10:21:43,899][INFO ][o.e.t.TransportService   ] [lab-elk-1] publish_address {127.0.0.1:9300}, bound_addresses {[::]:9300}
[2025-03-25T10:21:44,088][INFO ][o.e.b.BootstrapChecks    ] [lab-elk-1] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2025-03-25T10:21:44,093][INFO ][o.e.c.c.ClusterBootstrapService] [lab-elk-1] this node has not joined a bootstrapped cluster yet; [cluster.initial_master_nodes] is set to [[master-node-ipv6]]
[2025-03-25T10:21:44,099][INFO ][o.e.c.c.ClusterBootstrapService] [lab-elk-1] skipping cluster bootstrapping as local node does not match bootstrap requirements: [[master-node-ipv6]]
[2025-03-25T10:21:44,913][INFO ][o.e.c.c.JoinHelper       ] [lab-elk-1] failed to join {master-node}{QMruOrUFSZS_wCuvsnT8BQ}{89tCkQwiQziGxsFOq_HuCg}
...

elasticsearch.yml from the second node:

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: my-cluster
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
#
# Path to log files:
#
#path.logs: /path/to/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
#network.host: 192.168.0.1
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["[master-node-ipv6]"]
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
#action.destructive_requires_name: false

#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically
# generated to configure Elasticsearch security features on 20-03-2025 10:34:40
#
# --------------------------------------------------------------------------------

#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically
# generated to configure Elasticsearch security features on 25-03-2025 09:12:50
#
# --------------------------------------------------------------------------------

# Enable security features
xpack.security.enabled: true

xpack.security.enrollment.enabled: true

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
# Discover existing nodes in the cluster
discovery.seed_hosts: ["[mastern-node-ipv6]:9300"]

# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0

# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
transport.host: 0.0.0.0

#----------------------- END SECURITY AUTO CONFIGURATION -------------------------
18

Think I found it....

The initial master nodes names needs to match exactly what is in the first elasticsearch.yml

I think this all comes from you initially manually setting the node name on the initial node which does not match the host name.

See here for details from the docs

Try commenting out the node name in the first elasticsearch.yml ... In short, don't set the node name unless you really know what you're doing let that be done automatically

Restart The first node.

Then do a clean install and token on the second.

I should have seen / asked this earlier...

19

There’s been a few threads like this since I started following the forum.

This is this comment: above the setting:

Maybe I’m in a minority, or too old school, but I don’t think that’s the most helpful suggestion, certainly not in every scenario.

Whenever I’m clustering sonething , be that ELK or any of 101 other technologie, one if the first things I’m considering is how does each node know the “name” of every other node, including itself, in a consistent way. Forward and reverse resolutions are part of that, so that communication can be clearly defined.

That said, wording a suggestion to cover all scenarios is also hard.

20

I've tried doing what you suggested but I keep getting the same error as before.

I made sure to always delete the data folder as well but to no avail. This is the current state of the elasticsearch.yml on both nodes after running the enrollment command on the second node:

first node:

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: master-node-hostname
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: master-node
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/opt/elasticsearch/data
#
# Path to log files:
#
path.logs: /var/opt/elasticsearch/log
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
#network.host: 0.0.0.0
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: [""]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
#action.destructive_requires_name: false

#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically
# generated to configure Elasticsearch security features on 21-06-2024 08:07:18
#
# --------------------------------------------------------------------------------

# Enable security features
xpack.security.enabled: true

xpack.security.enrollment.enabled: true

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
cluster.initial_master_nodes: ["master-node-hostname"]

# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: ["[master-node-ipv6]"]

# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
transport.host: ["[master-node-ipv6"]

#----------------------- END SECURITY AUTO CONFIGURATION -------------------------

second node:

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: master-node-hostname
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
#
# Path to log files:
#
#path.logs: /path/to/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
#network.host: 192.168.0.1
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:

#cluster.initial_master_nodes: [""]


# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
#action.destructive_requires_name: false

#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically
# generated to configure Elasticsearch security features on 25-03-2025 16:22:49
#
# --------------------------------------------------------------------------------

# Enable security features
xpack.security.enabled: true

xpack.security.enrollment.enabled: true

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
# Discover existing nodes in the cluster
discovery.seed_hosts: ["[master-node-ipv6]:9300"]

# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0

# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
transport.host: 0.0.0.0

#----------------------- END SECURITY AUTO CONFIGURATION -------------------------

I've also tried commenting and uncommenting the discovery.seed_hosts and cluster.initial_master_nodes options and restarting but I get the same errors as before.

Also, thank you so much for your help so far and for your patience!!