Manually Add node to cluster Elasticsearch 8.6

[syifelastic (DT)]

Hello. I have a 3 node Elasticsearch cluster. I originally set up the 3 nodes with an enrollment token. However, I later changed from http keystore to a certificate/key configuration in the yml.

This breaks the enrollment token function. The transport.p12 remains the same with keystore intact.

How can I go about adding a node without changing from the certificate/key xpack config?

The end state to move to a 6 node cluster with 3 nodes in one data center and 3 in another.

The change from HTTP keystore was necessary for the application that is riding on top of the elastic database.

[leandrojmp (Leandro Pereira)]

Can you share the elasticsearch.yml of your nodes?

Never used this enrollment token, but to manually a node to an existing cluster you basically just need to configure the discovery.seed_hosts of the new node with the master nodes, and configure the path for both the node certificates, transport certificates and certificate authorities.

What is in the elasticsearch.yml and what is in the keystore?

@syifelastic

The end state to move to a 6 node cluster with 3 nodes in one data center and 3 in another.

What would be the reason to have a cluster like that? Where your master nodes will be allocated? If I'm not wrong, this configuration is not recommended as the latency between the data centers can impact the cluster.

[syifelastic (DT)]

I was looking at the discovery seed hosts. Does the transport.P12 have to be migrated over as well? Or will does it get discovered in the bootstrap process?

I will try this tomorrow with the discovery seed hosts.

I can't share my .Yml today as I am now out of office. Will update tomorrow.

And for the arrangement of the nodes, we want to put the three new nodes in a non-master eligible configuration and only promote if something goes bad in the original cluster.

The application riding on top of elastic explicitly does not support many of the high availability options that elasticsearch does. So the expansion of the cluster is the only option we have to satisfy our requirements

[syifelastic (DT)]

@leandrojmp

What is in the elasticsearch.yml and what is in the keystore?

The key store is only active for the transport. With the key passphrase set with the ./elasticsearch-keystore add xpack.Security.key.secure_passphrase

[leandrojmp (Leandro Pereira)]

@syifelastic

Does the transport.P12 have to be migrated over as well? Or will does it get discovered in the bootstrap process?

Nothing is migrated, you will need to have a transport certificate to the new nodes.

I'm not sure how this enrollment token process works because I never used it.

But in the end you just need a transport certificate for every new node signed by the same certificate authority, if they are stored on elasticsearch-keystore or on some file does not make much difference.

Do you have the CA to sign new nodes certificates? If yes, than you just need to create the certificates, you can follow the documentation for elasticsearch-certutil here.

@syifelastic

And for the arrangement of the nodes, we want to put the three new nodes in a non-master eligible configuration and only promote if something goes bad in the original cluster.

Yeah, this does not work, if the datacenter with the 3 non-master nodes lose connectivity with the datacenter with the master nodes, you won't be able to access the data on those 3 non-master nodes and you also can not change them to be masters.