Bug? "Could not communicate with the node... from the enrollment token."

[smm (Stefano)]

Hi there,
I am making my first steps in Elasticsearch 8.x coming from 7.x.
On 2 local virtual box nodes I would like to test / to understand the procedure of how I can let join a second Elasticsearch node to an already one-node cluster with Elasticsearch & kibana on it.
Problem: They do not find to each other. I tried several ideas, but it did not work out. Do you have a hint for me?

Some details:
Running on ubuntu 20.04
Installing via vagrant / ansible provisioner
Using: 8.2.0 for both nodes
Already running fine is this node: AllInOne820redis2v4
IP: 192.168.56.113

Here I would like to install / joint Elasticsearch: InOne820redis1v4
IP: 192.168.56.112

I did this on the already fine running node :
/usr/share/Elasticsearch/bin/Elasticsearch-create-enrollment-token -s node

I attach the Elasticsearch.yml of the running node.
On the-to-be-attched node I still need to do a apt-get install for the Elasticsearch deb package.

My question: What steps do I have to do on both nodes so that allyInOne820redis1v4 joins the already fine running one-node-cluster?

Thank you!!

root@AllInOne820redis2v4:/etc/elasticsearch# cat elasticsearch.yml
cluster.name: AllInOne820redis2
node.name: AllInOne820redis2
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 192.168.56.113
action.destructive_requires_name: true
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
http.host: 0.0.0.0
discovery.type: single-node
cluster.routing.allocation.disk.threshold_enabled: false

The un-joined Elasticsearch.yml looks like this:

root@AllInOne820redis2v4:/etc/elasticsearch# cat /etc/elasticsearch/elasticsearch.yml 
cluster.name: AllInOne820redis2 
node.name: AllInOne820redis1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
bootstrap.memory_lock: true
network.host: 192.168.56.112
discovery.seed_hosts: ["192.168.56.113"]
action.destructive_requires_name: true
xpack.security.enabled: true
xpack.security.enrollment.enabled: true
xpack.security.http.ssl:
  enabled: true
  keystore.path: certs/http.p12
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
http.host: 0.0.0.0
discovery.type: single-node
cluster.routing.allocation.disk.threshold_enabled: false

[smm (Stefano)]

I am at this point that I get this error message:

root@AllInOne820redis1v4:/tmp# /usr/share/Elasticsearch/bin/Elasticsearch-reconfigure-node --enrollment-token eyJ2ZXIiOiI4LjIuMCIsImFkciI6WyIxMC4wLjIuMTU6OTIwMCJdLCJmZ3IiOiI3ODgzZjlhZWUyNzNhZGQyMTMzZWMwZDQ5NmViMTlkMzgyZDI5YmQ0YzI3MTc1ZDVkODc4ZDgwN2FlN2QzNzM0Iiwia2V5IjoiZkptWEk0RUJTcVg2X0xJOGQtNGk6QWhJSjhWaUVRYy0wWHdCeUZUMERDdyJ9

This node will be reconfigured to join an existing cluster, using the enrollment token that you provided.
*This operation will overwrite the existing configuration. Specifically: *

• • Security auto configuration will be removed from Elasticsearch.yml*
• • The [certs] config directory will be removed*
• • Security auto configuration related secure settings will be removed from the Elasticsearch.keystore*
    Do you want to continue with the reconfiguration process [y/N]y
    Unable to communicate with the node on https://10.0.2.15:9200/_security/enroll/node. Error was Connection refused
    ERROR: Aborting enrolling to cluster. Could not communicate with the node on any of the addresses from the enrollment token. All of [10.0.2.15:9200] were attempted.

This is logical since it should look at this IP 192.168.56.113 where the second node is running.

QUESTION: Is there a possibility to tell Elasticsearch-reconfigure-node to use a specific IP / network card?

Thanks!!

[ibra_013 (Ibra13)]

Hi @smm

take a look at this

Add and remove nodes in your cluster | Elasticsearch Guide [8.2] | Elastic

Important Elasticsearch configuration | Elasticsearch Guide [8.2] | Elastic

[smm (Stefano)]

Sorry, I have already taken into account the links.

The point is: on the machine there are 2 IP adresses and this command:
/usr/share/Elasticsearch/bin/Elasticsearch-reconfigure-node --enrollment-token...
takes into consideration the false IP.

Therefore I get this error:
Could not communicate with the node on any of the addresses from the enrollment token. All of [10.0.2.15:9200] were attempted.

So: How can I tell Elasticsearch-reconfigure-node to use the right IP 192.168.56.112?
Cheers

[smm (Stefano)]

I did this:
/usr/share/Elasticsearch/bin/Elasticsearch-keystore show xpack.security.http.ssl.keystore.secure_password
keytool -keystore /etc/Elasticsearch/certs/http.p12 -storepass <password_I_got_above> -list -v

What I see is:
SubjectAlternativeName [
DNSName: localhost
IPAddress: 192.168.56.112
DNSName: AllInOne820redis1v4
IPAddress: 0:0:0:0:0:0:0:1
IPAddress: 127.0.0.1
IPAddress: fe80:0:0:0:47:98ff:fef4:3985
IPAddress: fe80:0:0:0:a00:27ff:fec1:a7c2
IPAddress: 10.0.2.15
This means that all IPs are known. So why does it say: All of [10.0.2.15:9200] were attempted? Only 1 IP was attampted. Is this a bug?

[ikakavas (Ioannis Kakavas)]

Did you generate the enrollment token after you added network.host: 192.168.56.112 to the configuration and restarted the node ? The create-enrollment-token tool will connect to the existing node to get information about the node, it will not read the configuration file, so if you made that change but did not restart the node, these changes will not be taken into consideration.

Also, FYI, it looks like you have manually modified the second node configuration file after installation before you ran elasticsearch-reconfigure-node. If you do so, please add configuration options without changing the format of the file. We support running this tool directly after installation, and while we will be permissive in added config options, but we expect to find certain sections in place in the file we have just generated.

[smm (Stefano)]

Thank you for replying! - This is what I re-did now:
On AllInOne820redis2v4 / 192.168.56.113:

• rebooted the all-in.one running / working node with the IP 192.168.56.113
• Still want to add a second node to 192.168.56.113
• /usr/share/Elasticsearch/bin/Elasticsearch-create-enrollment-token -s node

Then on the newly created to-be-added node with the IP 192.168.56.112 I did:

• vim /etc/hosts
  --> Added 2 lines
  192.168.56.112 AllInOne820redis1v4
  192.168.56.113 AllInOne820redis2v4
• apt-get install /tmp/Elasticsearch.deb
• vim /etc/Elasticsearch/Elasticsearch.yml
  --> here I added 3 lines:
  cluster.name: AllInOne820redis2
  node.name: AllInOne820redis1
  network.host: 192.168.56.112
  (no other changes made)
  (Elasticsearch has NOT initially been started)
• /usr/share/Elasticsearch/bin/Elasticsearch-reconfigure-node --enrollment-token >THE KEY from node .113<

RESULTS:
Unable to communicate with the node on https://10.0.2.15:9200/_security/enroll/node. Error was Connection refused
ERROR: Aborting enrolling to cluster. Could not communicate with the node on any of the addresses from the enrollment token. All of [10.0.2.15:9200] were attempted.

Any idea left what I can further test?
Thank you!!

[smm (Stefano)]

The soution is:
"It seems like this is caused by the http.host: 0.0.0.0 setting in master's config file. They recommend commenting this setting out, which should trigger the server to take network.host setting into account."