How do you bootstrap an elastic cluster for production with security enabled? The documentation feels unclear

[Atharv_Kirtikar]

I am trying to create a 3 node cluster for production.

The documentation here implies that with security creating an enrolment token is the way to go.

However it also recommends here that you should have at least three nodes set up to bootstrap in production:

It is technically sufficient to set cluster.initial_master_nodes on a single master-eligible node in the cluster, and only to mention that single node in the setting’s value, but this provides no fault tolerance before the cluster has fully formed. It is therefore better to bootstrap using at least three master-eligible nodes, each with a cluster.initial_master_nodes setting containing all three nodes.

Having separate security certificates on startup, however, means that simply starting up three nodes doesn't let them find each other.

Simply setting up one node manually and then the other two nodes using the token doesn't seem to work either - if the initial master node restarts it fails to locate the rest of the cluster. On the other hand, if you modify elasticsearch.yml too much on a node before enrolling it just refuses to enroll.

How do I set up a resilient and secure 3 node cluster? I've spent ages on this and am lost, would really appreciate any help.

[warkolm (Mark Walkom)]

Welcome to our community!

Generally what you do is install Elasticsearch on a node, setup all the certificates etc, start it, install Elasticsearch on other nodes, copy the certs etc to them, then add that node.

Add and remove nodes in your cluster | Elasticsearch Guide [8.11] | Elastic does go into it more;

1. Set up a new Elasticsearch instance.
2. Specify the name of the cluster with the cluster.name setting in elasticsearch.yml. For example, to add a node to the logging-prod cluster, add the line cluster.name: "logging-prod" to elasticsearch.yml.
3. Start Elasticsearch. The node automatically discovers and joins the specified cluster.

To add a node to a cluster running on multiple machines, you must also set discovery.seed_hosts so that the new node can discover the rest of its cluster.

[Atharv_Kirtikar]

Thanks for answering @warkolm!

Sadly, the series of steps you've linked don't seem to talk about setting up a node to join a cluster with certificates/security, which is unfortunate considering it is enabled by default in the current Elasticsearch version (8.3.2).

After spending a lot of time reading through docs (and a lot of testing), I think I've figured out my confusion though.
Please correct me if I'm wrong with any of my assumptions.

1. Using an enrolment token to make a node "join" a cluster - all it really does is automatically copy certificates over to a brand new node.

2. Once the certificates have been added, you can still modify the elasticsearch.yml file on the new node before starting it. (This is important because if you modify the the .yml file before using the enrolment token, enrolling doesn't work and you're forced to reinstall elasticsearch.)

3. Once the node is started, it uses the elasticsearch.yml configuration and the transferred SSL certificates to connect to the cluster.

If I'm right about this, I really feel there should be a more detailed/prominent description of exactly what elasticsearch-reconfigure-node does and how it should be used to set up a production cluster, because right now it feels like a finicky black box that forces you to start all over again if you do anything wrong.