Adding a node to a cluster , using 3 hosts, AlmaLinux 9, Elasticsearch 8.17

Elastic Stack Elasticsearch
1

Hello, I am quite new to Elasticsearch and having trouble adding nodes to my single node cluster. All nodes are on their own servers. Most responses on this subject are several years old. So Im asking again.

My first node was downloaded using the tar.gz.
I referenced Add and remove nodes in your cluster | Elasticsearch Guide [8.17] | Elastic
I configured the elasticsearch.yml thusly:

cluster-name:  my-application
   node.name: <hostname>
   network.host: 0.0.0.0
   discovery.seed_hosts:[ "<hostname-1>","<hostname-2>","<hostname-3>"]

I ran bin/elastisearch

The security auto configuration was generated once I started the single node cluster. I can list those configs if needed later.

I generated an enrollment token

On my second machine(aka second node), I ran

bin\elasticsearch --enrollment-token <enrollment-token>

The second node does not start instead it reports

ERROR:Skipping security auto configurationbecause, this node is confiugred to bootstrap or to join a multi-node cluster, which is not supported., with exit code 80

I made no changes to the 2nd nodes yaml as the instructions did not say to do so. I obvously did not follow the process correctly and would appreciate any insights.
kind regards.

2

Hi @Trent-alex Welcome to the community.

Per the docs you referenced you need to bund the transport layer to the network.

On the first node... Set and restart

transport.host: 0.0.0.0

Also set the network.host , transport.host, and discovery.seed_hosts on the 2nd node

Then try again

Additionally, only nodes on the same host can join the cluster without additional configuration. If you want nodes from another host to join your cluster, you need to set transport.host to a supported value (such as uncommenting the suggested value of 0.0.0.0), or an IP address that’s bound to an interface where other hosts can reach it. Refer to transport settings for more information.

Then.... Try again on the 2nd node, you may need to regenerate the enrollment token

3

Thank you for your response.
THe 1st node's yaml has the following changes:

cluster.name: <cluster>
node.name: <hostname>
network.host: 0.0.0.0
discovery.seed_hosts:[ "<hostname-1>","<hostname-2>","<hostname-3>"]
transport.host: 0.0.0.0

Node 1 has a warning summaried as thus: THis node is a fully-formed single node cluster, but it is configured as if to discover other nodes and form a multi-node cluster via discovery.seed_hosts. Remove the discovery configuration to suppress this message

Node 2's yml is:

cluster.name: <same as node 1>
network.host: 0.0.0.0
discovery.seed_hosts:  < same as node 1>
transport.host: 0.0.0.0

Node 2 is generatng:

ERROR:Skipping security auto configurationbecause, this node is confiugred to bootstrap or to join a multi-node cluster, which is not supported., with exit code 80

I believe I addressed your changes. I appreciate any additional insight.

4

What other settings are set in your In your first node, can you please share the entire elasticsearch.yml

This implies you have single-node-discovery set

Actually, please share both .ymls

5

I am actively correcting this is state now.
Node 1

# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: her
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
#
# Path to log files:
#
#path.logs: /path/to/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: her
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["her","zel","gar"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["her"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:

Node 2

# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: zel
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
#
# Path to log files:
#
#path.logs: /path/to/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: zel
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["her","zel", "gar"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["her"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
6

With the above yml configs node 1 is still reporting that is a fully formed single -node cluster. First, Im not certain that this precludes adding more nodes. Second, if it does what configuration am I missing to stop this behavior

7

Those are not the entire ymls...

transport.host is not set...

also please show the actual messages not your summary of them... The more accurate and complete information you provide the quicker we can help..

8

According to the docs for this setting this is not good advice:

Use this setting only if you require different configurations for the transport and HTTP interfaces.

No, it doesn't. It means it's a single-node cluster with a discovery config, in this case, discovery.seed_hosts, which won't do anything on this node since the cluster already formed.

This should not be set, according to the docs for this setting:

Do not configure this setting on nodes joining an existing cluster.

You need to either use --enrollment-token to configure this node automatically, or else you must configure it entirely manually. You are apparently trying to do both, and ES is getting confused.

9

Thank you both for you added insight. Nodes 1yaml as stated above, transport.host was not changed

# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: her
#
# Add custom attributes to the node:
#
#node.attr.rack: r1
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
#
# Path to log files:
#
#path.logs: /path/to/logs
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
#bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
#
network.host: her
#
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
#
#http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["her", "zel","gar"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["her"]
#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Allow wildcard deletion of indices:
#
#action.destructive_requires_name: false

#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically      
# generated to configure Elasticsearch security features on 27-12-2024 21:30:17
#
# --------------------------------------------------------------------------------

# Enable security features
xpack.security.enabled: true

xpack.security.enrollment.enabled: true

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
#----------------------- END SECURITY AUTO CONFIGURATION -------------------------

It is producing this warning

2024-12-27T15:33:38,703][WARN ][o.e.c.c.Coordinator      ] [her] This node is a fully-formed single-node cluster with cluster UUID [s_l_SZtFQ------------------------], but it is configured as if to discover other nodes and form a multi-node cluster via the [discovery.seed_hosts=[her, zel, gar]] setting. Fully-formed clusters do not attempt to discover other nodes, and nodes with different cluster UUIDs cannot belong to the same cluster. The cluster UUID persists across restarts and can only be changed by deleting the contents of the node's data path(s). Remove the discovery configuration to suppress this message. See [https://www.elastic.co/guide/en/elasticsearch/reference/8.17/modules-discovery-bootstrap-cluster.html#modules-discovery-bootstrap-cluster-joining] for more information
10

Node 2 after using enrollment token reports

[admin@zel elasticsearch-8.17.0]$ bin/elasticsearch --enrollment-token eyJ2ZXIiOiI4LjE0LjAiLCJhZHIiOlsiMTkyLjE2OC4xMC42MDo5MjAwIl0sImZnciI6IjNlYTVlOGZhYTM3ZmEzZjYzYWJjMTllM2M3MDQxZjFlNWMzOTRjMDk3ZDE5ZTQ3ZDkzNTZlY2Y1YzRhYmE0MDIiLCJrZXkiOiJqSEUxQ3BRQkhNUzJrdlh0RHk5OTpJeDE0XzkyblFnYUVkbnRjbkNtTE9BIn0=

ERROR: Skipping security auto configuration because this node is configured to bootstrap or to join a multi-node cluster, which is not supported., with exit code 80
[admin@zel elasticsearch-8.17.0]$
11

Thanks @DavidTurner for squaring me on that.... Guess I am just full of bad advice today... Ughh.. You think I would keep all this straight... After all these years...

1 Like
12

You need to either use --enrollment-token to configure this node automatically, or else you must configure it entirely manually. You are apparently trying to do both, and ES is getting confused. (repeating my earlier message)

13

@DavidTurner So Here is my confusion... after running

./bin/elasticsearch

On an OOTB tar.gz you get these instructions

• On this node:
  ⁃ Create an enrollment token with `bin/elasticsearch-create-enrollment-token -s node`.
  ⁃ >>> Uncomment the transport.host <<< setting at the end of config/elasticsearch.yml.
  ⁃ Restart Elasticsearch.
• On other nodes:
  ⁃ Start Elasticsearch with `bin/elasticsearch --enrollment-token <token>`, using the enrollment token that you generated.

And the last lines in the config. And if you want to form a cluster that is not on localhost, you have to uncomment the last line.

# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0

# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
#transport.host: 0.0.0.0

That is why I remembered to set transport host ... so our docs may be a bit confusing...

@Trent-alex

What David is saying is that you are trying to both
manually configure your clusters and
trying auto-enrollment
that is why this is not working...

So what I just did on 2 computers.... for a test.

Downloaded and untarred the distros

Did not edit anything...

One the first node I ran

./bin/elasticsearch

Then, followed these directions...without editing anything but uncommenting the transport.host on the first node

Then, without editing anything on the second node, I ran the start with the enrollment token.

ℹ️  Configure other nodes to join this cluster:
• On this node:
  ⁃ Create an enrollment token with `bin/elasticsearch-create-enrollment-token -s node`.
  ⁃ Uncomment the transport.host setting at the end of config/elasticsearch.yml.
  ⁃ Restart Elasticsearch.
• On other nodes:
  ⁃ Start Elasticsearch with `bin/elasticsearch --enrollment-token <token>`, using the enrollment token that you generated.

And everything worked just fine... the cluster formed.

If you want to set up everything manually ... which is good to learn... do not use the enrollment token method. You will need to learn more about discovery and cluster formation in detail

14

As someone who is still new, I believe using the enrollment token is the easiest way. I had some issues and redid mine and it worked. I have been working on it for a few days and after some help on the elasticsearch subreddit I got it working with the token.

15

Thank you all for you responses. I was able to add a second cluster. I will summarize my steps, hopefully this help the next person. Please note following occured before @stephenb last post but you can see the commonalities.

My primary references are:
Add and remove nodes in your cluster | Elasticsearch Guide [8.17] | Elastic

On Node 1 I changed the following in the elasticsearch.yml

cluster.name:  < clustername> # consider using elasticsearch as this is the default name
node.name:  node-1
network.host: <ip address or hostname>
cluster.initial_master_nodes: ["node-1","<other nodes>"] 
# Im not certain this is needed and there is a warning about leaving it in place here: https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-discovery-settings.html
http.host: 0.0.0.0 
# I believe this conflicts with the reference but is needed if using different hosts

Run bin/elasticsearch
in another terminal get a new enrollment token if previous is older than 30 min

On second host soon to be 2nd Node
:reminder_ribbon:remember to open ports 9200 & 9300,
Make NO changes to the elasticsearch.yml

bin\elasticsearch --enrollment-token <enrollment-token>

After successfull enrollment, I changed the node.name of the second node

1 Like