Need help setting up a second node

Elastic Stack Elasticsearch
1

Hello!

I have a server with Elasticsearch, Kibana and a fleet-server agent installed.
I use all the default certificates created in the beginning (automatic security configuration I believe it's called?).

I would like to add a second server with Elasticsearch (a second node). I've tried using the enrollment token but it says it can't reach the node:

Unable to communicate with the node on ://127.0.0.1:9200/_security/enroll/node. Error was Connection refused
ERROR: Aborting enrolling to cluster. Could not communicate with the node on any of the addresses from the enrollment token. All of [127.0.0.1:9200] were attempted., with exit code 69

What do I need to do for this enrollment token to work? Is there another way to enroll the node?

Every help and piece of information is welcome!

Thanks!

2

Hi @hairless_mess Welcome to the community.

We will need some basics to help

What documentation did you follow?

What Version?

How did you install elasticsearch?

Please show the full command(s) you used for the enrollment process (creating the token, applying the token)

Share the elasticsearch.yml for both nodes?

Perhaps with that we can help.

3

Hello!

I followed this documentation:

I'm using version 8.17.0

I deployed it using the .tar package

On the master node:
elasticsearch-create-enrollment-token -s node

On the new node:
elasticsearch --enrollment-token <enrollment-token>

This is the config for my master node:

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: my-machine
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: master-node
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
path.data: /var/opt/elasticsearch/data
#
# Path to log files:
#
path.logs: /var/opt/elasticsearch/log
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
#network.host: 0.0.0.0
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: [""]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
#action.destructive_requires_name: false

#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically
# generated to configure Elasticsearch security features on 21-06-2024 08:07:18
#
# --------------------------------------------------------------------------------

# Enable security features
xpack.security.enabled: true

xpack.security.enrollment.enabled: true

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
cluster.initial_master_nodes: ["my-machine"]

# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0

# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
#transport.host: 0.0.0.0

#----------------------- END SECURITY AUTO CONFIGURATION -------------------------

This is for the second node:

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
#
# Path to log files:
#
#path.logs: /path/to/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
#network.host: 192.168.0.1
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["node-1", "node-2"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
#action.destructive_requires_name: false
4

Notice at the bottom of the first .yml

Uncomment the transport.host: 0.0.0.0

# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
#transport.host: 0.0.0.0

I would review this section

Additionally, only nodes on the same host can join the cluster without additional configuration. If you want nodes from another host to join your cluster, you need to set transport.host to a supported value (such as uncommenting the suggested value of 0.0.0.0), or an IP address that’s bound to an interface where other hosts can reach it. Refer to transport settings for more information.

5

After uncommenting the line, restarting elasticsearch, recreating the enrollment, and trying again to enroll the node, I get the same error.

6

Please share the exact command you ran with the exact error message.

7

./elasticsearch --enrollment-token XXXXXX==

Unable to communicate with the node on https://127.0.0.1:9200/_security/enroll/node. Error was Connection refused
ERROR: Aborting enrolling to cluster. Could not communicate with the node on any of the addresses from the enrollment token. All of [127.0.0.1:9200] were attempted., with exit code 69
8

Hmmmm

On the initial node Can you try putting the IP address of that machine in and try the restart regenerate etc...

http.host: 192.168.10.1
transport.host:  192.168.10.1

Also as a check from the second machine

curl -v -k -u elastic https://<ip_of_initial>:9200

9

I think it worked, how can I check if the node joined the cluster?

I had to change some little things here and there, I will post everything when I'm done.

10

Either of these

GET  _cat/nodes?v
GET _cluster/health
11

Not working yet. When I run the enroll command I get this at the end:

[2025-03-19T17:37:32,075][INFO ][o.e.c.c.ClusterBootstrapService] [second-node] this node has not joined a bootstrapped cluster yet; [cluster.initial_master_nodes] is set to []
[2025-03-19T17:37:32,925][INFO ][o.e.c.c.JoinHelper       ] [second-node] failed to join {master-node}{QMruOrUFSZS_wCuvsnT8BQ}{89tCkQwiQziGxsFOq_HuCg}{master-node}{master-node-IP}{[master-node-IP]:9300}{cdfhilmrstw}{8.17.0}{7000099-8521000}{ml.max_jvm_size=4148166656, ml.config_version=12.0.0, xpack.installed=true, transform.config_version=10.0.0, ml.machine_memory=8290643968, ml.allocated_processors=2, ml.allocated_processors_double=2.0} with JoinRequest{sourceNode={second-node}{umreEYrATDGOhfWw6FJ2tQ}{xdfoDXgmRuyEtd71dyl_Qw}{second-node}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw}{8.17.0}{7000099-8521000}{ml.max_jvm_size=4148166656, ml.config_version=12.0.0, xpack.installed=true, transform.config_version=10.0.0, ml.machine_memory=8290635776, ml.allocated_processors=2, ml.allocated_processors_double=2.0}, compatibilityVersions=CompatibilityVersions[transportVersion=8797002, systemIndexMappingsVersion={.secrets-inference=MappingsVersion[version=1, hash=-1434574148], .triggered_watches=MappingsVersion[version=1, hash=-502826165], .fleet-agents-7=MappingsVersion[version=1, hash=-270798539], .fleet-servers-7=MappingsVersion[version=1, hash=-916922632], .fleet-policies-leader-7=MappingsVersion[version=1, hash=-1108172796], .ml-config=MappingsVersion[version=1, hash=-319778629], .security-tokens-7=MappingsVersion[version=1, hash=576296021], .geoip_databases=MappingsVersion[version=1, hash=-305757839], .snapshot-blob-cache=MappingsVersion[version=1, hash=632712485], .security-profile-8=MappingsVersion[version=2, hash=-909540896], .search-app-1=MappingsVersion[version=1, hash=-501711141], .watches=MappingsVersion[version=1, hash=-1045118511], .fleet-artifacts-7=MappingsVersion[version=1, hash=-1593703898], .query-rules-2=MappingsVersion[version=2, hash=-1272560824], .transform-internal-007=MappingsVersion[version=1, hash=1144737897], .fleet-enrollment-api-keys-7=MappingsVersion[version=1, hash=-1804942283], .fleet-actions-7=MappingsVersion[version=1, hash=-2624357], .tasks=MappingsVersion[version=0, hash=-945584329], .fleet-secrets-7=MappingsVersion[version=1, hash=-745394230], .ml-meta=MappingsVersion[version=2, hash=-613742866], .security-7=MappingsVersion[version=3, hash=-832976091], .connector-secrets-1=MappingsVersion[version=1, hash=-745394230], .logstash=MappingsVersion[version=1, hash=-1058806351], .ml-inference-000005=MappingsVersion[version=3, hash=919553140], .async-search=MappingsVersion[version=0, hash=-1403744380], .inference=MappingsVersion[version=2, hash=-1459421596], .fleet-policies-7=MappingsVersion[version=1, hash=-201702522], .synonyms-2=MappingsVersion[version=1, hash=-888080772]}], features=[simulate.mapping.addition, mapper.vectors.bbq, text_similarity_reranker_retriever_composition_supported, esql.casting_operator, standard_retriever_supported, flattened.ignore_above_support, features_supported, stats.include_disk_thresholds, mapper.ignore_above_index_level_setting, data_stream.rollover.lazy, routing.multi_value_routing_path, get_database_configuration_action.multi_node, desired_node.version_deprecated, search.vectors.k_param_supported, mapper.source.synthetic_source_with_copy_to_and_doc_values_false, mapper.source.synthetic_source_fallback, mapper.range.null_values_off_by_one_fix, mapper.keyword_normalizer_synthetic_source, mapper.subobjects_auto, mapper.keyword_dimension_ignore_above, script.term_stats, esql.mv_sort, esql.mv_ordering_sorted_ascending, health.dsl.info, esql.spatial_points_from_source, mapper.query_index_mode, esql.st_x_y, esql.base64_decode_encode, routing.boolean_routing_path, mapper.pass_through_priority, esql.async_query, unified_highlighter_matched_fields, esql.agg_values, geoip.downloader.database.configuration, mapper.track_ignored_source, retrievers_supported, put_database_configuration_action.ipinfo, rest.local_only_capabilities, text_similarity_reranker_retriever_supported, simulate.component.template.substitutions, esql.st_centroid_agg, query_rules.test, mapper.index_sorting_on_nested, security.roles_metadata_flattened, license-trial-independent-version, esql.disable_nullable_opts, script.hamming, rest.capabilities_action, mapper.source.synthetic_source_stored_fields_advance_fix, esql.resolve_fields_api, mapper.flattened.ignore_above_with_arrays_support, health.extended_repository_indicator, mapper.segment_level_fields_stats, esql.counter_types, knn_retriever_supported, mapper.subobjects_auto_fixes, mapper.vectors.bit_vectors, esql.spatial_shapes, data_stream.auto_sharding, esql.metrics_counter_fields, data_stream.lifecycle.global_retention, mapper.boolean_dimension, semantic_text.default_elser_2, logsdb_telemetry, usage.data_tiers.precalculate_stats, esql.st_intersects, random_reranker_retriever_supported, cluster.stats.source_modes, simulate.mapping.validation.templates, esql.st_contains_within, snapshot.repository_verify_integrity, esql.st_disjoint, rrf_retriever_composition_supported, mapper.source.synthetic_source_copy_to_fix, query_rule_retriever_supported, esql.timespan_abbreviations, mapper.synthetic_source_keep, query_rule_list_types, mapper.source.synthetic_source_copy_to_inside_objects_fix, tsdb.ts_routing_hash_doc_value_parse_byte_ref, logsdb_telemetry_stats, simulate.index.template.substitutions, file_settings, esql.string_literal_auto_casting, repositories.supports_usage_stats, rrf_retriever_supported, esql.from_options, mapper.vectors.int4_quantization, semantic_text.search_inference_id, esql.string_literal_auto_casting_extended, slm.interval_schedule, security.role_mapping_cleanup, security.migration_framework, simulate.mapping.validation, simulate.support.non.template.mapping, esql.metadata_fields], minimumTerm=37, optionalJoin=Optional[Join[votingNode={second-node}{umreEYrATDGOhfWw6FJ2tQ}{xdfoDXgmRuyEtd71dyl_Qw}{second-node}{127.0.0.1}{127.0.0.1:9300}{cdfhilmrstw}{8.17.0}{7000099-8521000}{ml.max_jvm_size=4148166656, ml.config_version=12.0.0, xpack.installed=true, transform.config_version=10.0.0, ml.machine_memory=8290635776, ml.allocated_processors=2, ml.allocated_processors_double=2.0}, masterCandidateNode={master-node}{QMruOrUFSZS_wCuvsnT8BQ}{89tCkQwiQziGxsFOq_HuCg}{master-node}{master-node-IP}{[master-node-IP]:9300}{cdfhilmrstw}{8.17.0}{7000099-8521000}{ml.max_jvm_size=4148166656, ml.config_version=12.0.0, xpack.installed=true, transform.config_version=10.0.0, ml.machine_memory=8290643968, ml.allocated_processors=2, ml.allocated_processors_double=2.0}, term=37, lastAcceptedTerm=0, lastAcceptedVersion=0]]}org.elasticsearch.transport.RemoteTransportException: [master-node][[master-node-ip]:9300][internal:cluster/coordination/join]
12

can you share the current state of the second nodes .yml

Unfortunately I think you are "half setup"

If you set this in the 2nd .yml

cluster.initial_master_nodes: <ip_of_first_node>

Start it I suspect the cluster will form.. but then you need to take that setting out... it is a 1 time setting see here

Then just try to resart the 2nd node without the enrollment token etc... i.e. just regular and see if you have the 2 nodes

13

This is my file after running the enroll command, without adding the cluster.initial_master_nodes:

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: my-cluster
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
#node.name: node-1
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
#
# Path to log files:
#
#path.logs: /path/to/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
#network.host: 192.168.0.1
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
#discovery.seed_hosts: ["host1", "host2"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
#cluster.initial_master_nodes: ["[myIP]"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
#action.destructive_requires_name: false

#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically
# generated to configure Elasticsearch security features on 20-03-2025 10:34:40
#
# --------------------------------------------------------------------------------

# Enable security features
xpack.security.enabled: true

xpack.security.enrollment.enabled: true

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
# Discover existing nodes in the cluster
discovery.seed_hosts: ["[myIP]:9300"]

# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0

# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
transport.host: 0.0.0.0

#----------------------- END SECURITY AUTO CONFIGURATION -------------------------

I tried doing like you said by adding the cluster.initial_master_nodes and restarting and then removing it and restarting again but it didn't work.

14

1st node you have

2nd Node you have

Those will not form a single cluster AFAIK as the cluster names are different.

What does that mean it did not work, please always include the diescription and error..

It did not work... the node did not start?

It did not work ... the node started but was not part of the cluster?

Change the cluster name in the 2nd put the master nodes in...

Start the cluster see if you see the 2 nodes... if so then remove the initial master node.

You may need to clean up the 2nd node (including data) and try to re-enroll

15

I have the same cluster name in both nodes, it was an error from my part when copying the configs here, my bad.

The node starts but it's not in the cluster.

I have done the steps you mention but the node still doesn't join the cluster.

I always restart from a clean snapshot of the machine before the enrollment.

16

Hmmm I am not sure what is going on...

You are following these instructions precisely? I have done this many times...

Including cleaning up the data directory. If you have not done that, this could explain a lot.

Are you still working with the initial node #1 or did you start over with that?

Can you share the logs where the cluster is forming from the 2nd node. This is important...

Can you share the current state of the 2 elasticsearch.yml.