# Problems Joining a Cluster **URL:** \ **Category:** Elasticsearch\ **Created:** [January 26, 2024, 11:33pm UTC](https://discuss.elastic.co/t/problems-joining-a-cluster/351904 "2024-01-26T23:33:28Z")\ **Posts on this page:** 14\ **Page:** 2
**Author:** ![bryanrood](https://avatars.discourse-cdn.com/v4/letter/b/ed8c4c/32.png) [@bryanrood](https://discuss.elastic.co/u/bryanrood)\ **Post date:** [January 30, 2024, 6:59pm UTC](https://discuss.elastic.co/t/problems-joining-a-cluster/351904/21 "2024-01-30T18:59:04Z")
Wow, so not using cluster.initial\_master\_nodes and using discovery.seed\_hosts worked! I did remove all the files in my ES data folder prior to starting just in case there was any residue from previous attempts. ES recreated all the files and this is the output of my cluster status now: ```auto { "cluster_name" : "clustername", "status" : "green", "timed_out" : false, "number_of_nodes" : 3, "number_of_data_nodes" : 3, "active_primary_shards" : 1, "active_shards" : 2, "relocating_shards" : 0, "initializing_shards" : 0, "unassigned_shards" : 0, "delayed_unassigned_shards" : 0, "number_of_pending_tasks" : 0, "number_of_in_flight_fetch" : 0, "task_max_waiting_in_queue_millis" : 0, "active_shards_percent_as_number" : 100.0 } ``` I restarted host1 and host4 took over as master! woohoo! ```auto ip heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name 10.2.1.46 0 27 0 0.00 0.02 0.00 cdfhilmrstw - host1 10.2.1.49 4 28 0 0.00 0.00 0.00 cdfhilmrstw * host4 10.2.1.47 2 28 0 0.00 0.01 0.00 cdfhilmrstw - host2 ``` So now, I'm onto adding 3 more nodes. I saw someone used the yml setting: node.roles [master, data] Is that valid to use to note which nodes can take over master? I also tried to set a cluster wide setting for replicas between the nodes and it failed. I may have to open a new topic for that. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [January 30, 2024, 7:17pm UTC](https://discuss.elastic.co/t/problems-joining-a-cluster/351904/22 "2024-01-30T19:17:28Z")
> [@bryanrood](#): > > Wow, > so not using cluster.initial\_master\_nodes and using discovery.seed\_hosts worked! Hurrah! > [@bryanrood](#): > > So now, I'm onto adding 3 more nodes. I saw someone used the yml setting: > node.roles [master, data] > Is that valid to use to note which nodes can take over master? I'd suggest you leave `node.roles` as the default (i.e. don't mention it in `elasticsearch.yml`) which means that all nodes have all roles. You only need set it if you have a pressing need for different nodes to have different roles, and if you've only got 6 nodes then they probably all want to do everything. ---
**Author:** ![bryanrood](https://avatars.discourse-cdn.com/v4/letter/b/ed8c4c/32.png) [@bryanrood](https://discuss.elastic.co/u/bryanrood)\ **Post date:** [January 30, 2024, 7:25pm UTC](https://discuss.elastic.co/t/problems-joining-a-cluster/351904/23 "2024-01-30T19:25:57Z")
David, Great suggestion. I split my 6 nodes between 2 cabinets, so that was my logic to have some multiple masters in another cabinet. A couple more questions for you. I noticed only 1 node says master. How do I get host2 and host4 to also be a master node at the same time? Or can only 1 be master at a time? When I add the other 3 nodes, should make the discovery.seed\_hosts have the original 3 that I wanted for master or add all hosts like this: ```auto discovery.seed_hosts: ["host1", "host2", "host4", "host3", "host5", "host6"] ``` Thanks, Bryan ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [January 30, 2024, 7:54pm UTC](https://discuss.elastic.co/t/problems-joining-a-cluster/351904/24 "2024-01-30T19:54:34Z")
> [@bryanrood](#): > > `cdfhilmrstw` The `m` in this alphabet soup means the nodes are all master-eligible. Only one of them is elected as master at once, but if it fails then another master-eligible one will take over. > [@bryanrood](#): > > When I add the other 3 nodes, should make the discovery.seed\_hosts have the original 3 that I wanted for master or add all hosts like this: `discovery.seed_hosts` should contain [the addresses of all the master-eligible nodes in the cluster](https://www.elastic.co/guide/en/elasticsearch/reference/current/important-settings.html#discovery-settings), so yeah mention all 6. It'll mostly work with fewer, at least as long as one or more of those three original nodes is still available. ---
**Author:** ![bryanrood](https://avatars.discourse-cdn.com/v4/letter/b/ed8c4c/32.png) [@bryanrood](https://discuss.elastic.co/u/bryanrood)\ **Post date:** [January 30, 2024, 11:11pm UTC](https://discuss.elastic.co/t/problems-joining-a-cluster/351904/25 "2024-01-30T23:11:20Z")
got it. I added the fourth node, following these steps and copying the cert file from the CA over and the node added to the cluster on the first try! I'm stoked. ---
**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\ **Post date:** [February 1, 2024, 12:51pm UTC](https://discuss.elastic.co/t/problems-joining-a-cluster/351904/26 "2024-02-01T12:51:40Z")
> [@bryanrood](#): > > Interestingly, on a brand new node, directly after install, prior to ever being run, the /etc/elasticsearch/certs folder is populated as is the /etc/elasticsearch/elasticsearch.keystore file. > > It's confusing why those files have certs/data in them at install time? I would think those should not populate until ES is run or the enrollment cmd populates them with data/certs from the cluster nodes. What do you think? RPM packages are expected to do all their installation configuration at install time, so the Elasticsearch installer configures security during package installation. The RPM install guide has a section on how to reconfigure the node to join an existing cluster. - [Install Elasticsearch with RPM | Elasticsearch Guide [8.12] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/rpm.html#_reconfigure_a_node_to_join_an_existing_cluster_2) ---
**Author:** ![bryanrood](https://avatars.discourse-cdn.com/v4/letter/b/ed8c4c/32.png) [@bryanrood](https://discuss.elastic.co/u/bryanrood)\ **Post date:** [February 1, 2024, 7:45pm UTC](https://discuss.elastic.co/t/problems-joining-a-cluster/351904/27 "2024-02-01T19:45:27Z")
Hi TimV, Thanks for checking this post out and responding. I followed the steps to a T multiple times on multiples and it just plain doesn't work. The enrollment-token method makes a bunch of assumptions that the document never covers, including the fact that none of the 4 steps indicates to modify the elasticsearch.yml to include the node name and cluster name which are a requirement to join a cluster. As I have documented above in this post, everytime I try to use the enrollment-token method in the link you posted I get the error that "...it appears that the node is not starting up for the first time...." These virgin, newly installed systems, there is no legacy install of ES anywhere. I would humbly request given your position in engineering that the documentation be updated with different steps and the other prerequisites that need to be completed prior to running ES for the first time with the enrollment token(such as file permissions and contents of the yml file). If you want to do a webex or screen share, I would be happy to show you it live. Thanks, Bryan ---
**Author:** ![bryanrood](https://avatars.discourse-cdn.com/v4/letter/b/ed8c4c/32.png) [@bryanrood](https://discuss.elastic.co/u/bryanrood)\ **Post date:** [February 1, 2024, 8:53pm UTC](https://discuss.elastic.co/t/problems-joining-a-cluster/351904/28 "2024-02-01T20:53:06Z")
I meant multiple times on multiple servers. ---
**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\ **Post date:** [February 1, 2024, 11:42pm UTC](https://discuss.elastic.co/t/problems-joining-a-cluster/351904/29 "2024-02-01T23:42:11Z")
As far as I can tell, you didn't follow the steps in that document You wrote > $ bin/elasticsearch --enrollment-token But that docs says to run ```auto /usr/share/elasticsearch/bin/elasticsearch-reconfigure-node --enrollment-token ``` Those are different commands. ---
**Author:** ![bryanrood](https://avatars.discourse-cdn.com/v4/letter/b/ed8c4c/32.png) [@bryanrood](https://discuss.elastic.co/u/bryanrood)\ **Post date:** [February 2, 2024, 12:30am UTC](https://discuss.elastic.co/t/problems-joining-a-cluster/351904/30 "2024-02-02T00:30:33Z")
Hi TimV, So your main guide: [Install Elasticsearch from archive on Linux or MacOS | Elasticsearch Guide [8.12] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/8.12/targz.html) Explicitly says halfway down under: "Enroll nodes in an existing cluster" to do this: From the installation directory of your new node, start Elasticsearch and pass the enrollment token with the `--enrollment-token` parameter. ```auto bin/elasticsearch --enrollment-token enrollment-token ``` Then there is this page: > **[Add and remove nodes in your cluster | Elasticsearch Guide \[8.12\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/8.12/add-elasticsearch-nodes.html)** that says to use: ```auto bin/elasticsearch --enrollment-token enrollment-token ``` So that's two current documents that say to use the elasticsearch binary. However, I saw that documentation with that option as well before I ever posted to this forum. So I started from scratch on a new server and ran the elasticsearch-reconfigure-node binary and it doesn't work. I spent a week trying to get that binary to work on a new node that had never been started. I would be happy to show you on a brand new system live. Thanks. ---
**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\ **Post date:** [February 2, 2024, 1:22am UTC](https://discuss.elastic.co/t/problems-joining-a-cluster/351904/31 "2024-02-02T01:22:13Z")
> [@bryanrood](#): > > So your main guide: [Install Elasticsearch from archive on Linux or MacOS | Elasticsearch Guide [8.12] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/8.12/targz.html) But you aren't installing from an archive. You downloaded the RPM, so you need to follow the instructions on the RPM page. You can't jump into a page about installing the `.tar.gz` package, skip the first few steps and expect the subsequent steps to work. I fully accept that it can be hard to determine which docs are relevant to your scenario, and many of the doc pages assume that you're starting from a particular point that might not be universal. The **Start the Elastic Stack with security enabled automatically** page provides the wrong information for people who install from RPM or deb packages. My goal here was to provide the background information for why the RPM install populates the certificates before first use, and how to overcome the error you were reporting > ERROR: Skipping security auto configuration because it appears that the node is not starting up for the first time. The node might already be part of a cluster and this auto setup utility is designed to configure Security for new clusters only. `reconfigure-node` is the step you need to apply if you use the RPM. If you ran into a problem with that then we can try to help, but we can't guess what steps you tried or what errors you encountered. ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [February 2, 2024, 6:02pm UTC](https://discuss.elastic.co/t/problems-joining-a-cluster/351904/32 "2024-02-02T18:02:01Z")
> [@TimV](#): > > The RPM install guide has a section on how to reconfigure the node to join an existing cluster. TIL, thanks Tim > [@TimV](#): > > The **Start the Elastic Stack with security enabled automatically** page provides the wrong information for people who install from RPM or deb packages ... and, worse, it doesn't mention that it only applies to archive installations. It just says this in its [prerequisites](https://www.elastic.co/guide/en/elasticsearch/reference/current/configuring-stack-security.html#_prerequisites_12), linking to a page that offers an RPM for download: > Download and unpack the elasticsearch package distribution for your environment. ---
**Author:** ![bryanrood](https://avatars.discourse-cdn.com/v4/letter/b/ed8c4c/32.png) [@bryanrood](https://discuss.elastic.co/u/bryanrood)\ **Post date:** [February 2, 2024, 8:03pm UTC](https://discuss.elastic.co/t/problems-joining-a-cluster/351904/33 "2024-02-02T20:03:04Z")
Thanks David and TimV, I appreciate you both trying to sort through this. This topic chain has definitely made the difference getting our production ES cluster online. As David mentions, it would help if the other documentation references to joining a cluster indicated both options, for RPM and non-RPM joins. It would also be helpful when running the reconfigure-node binary to indicate that the elasticsearch.yml still need to be modified with cluster name, network host, etc. The documentation gives the impression that it doesn't need to be modified. Maybe adding a note after the line "This node will be reconfigured to join an existing cluster, using the enrollment token that you provided. This operation will overwrite the existing configuration...." Thanks team! Bryan ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 1, 2024, 8:03pm UTC](https://discuss.elastic.co/t/problems-joining-a-cluster/351904/34 "2024-03-01T20:03:29Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed. [Previous page](https://discuss.elastic.co/t/problems-joining-a-cluster/351904.md?page=1)