# Can't connect node to cluster on another server

**URL:** <https://discuss.elastic.co/t/cant-connect-node-to-cluster-on-another-server/309432>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-security\
**Created:** [July 12, 2022, 2:41pm UTC](https://discuss.elastic.co/t/cant-connect-node-to-cluster-on-another-server/309432 "2022-07-12T14:41:08Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![dabit\_coder](https://avatars.discourse-cdn.com/v4/letter/d/91b2a8/32.png) [@dabit\_coder](https://discuss.elastic.co/u/dabit_coder)\
**Post date:** [July 12, 2022, 2:41pm UTC](https://discuss.elastic.co/t/cant-connect-node-to-cluster-on-another-server/309432/1 "2022-07-12T14:41:08Z")

</div>

First of all, I want to be clear that I looked at several guides and suggested similar questions before opening this post, but none of them worked for our case, so I am a bit desperate right now.

**Here is our situation:**  
The developer who created all this configuration is gone and we don't have a way to reach him. We don't have any password for the certificates or anything, in case that is needed.

Last week, our Elasticsearch stopped adding new records as it reached the state where the disk was almost full. We changed the config to get some time and now is working as expected. However, we want to add a new server with elasticsearch and form a cluster as we don't want to resize the disk because we don't want to lose anything.

Here is the configuration for the main server `/etc/elasticsearch/elasticsearch.yml`

```auto
cluster.name: goulue
node.name: master-node
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.max_content_length: 100mb
discovery.seed_hosts: ["127.0.0.1", "ip_node"]
xpack.license.self_generated.type: "basic"
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
#xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.certificate: "certs/master-node/master-node.crt"
xpack.security.transport.ssl.certificate_authorities: "certs/ca/ca.crt"
#xpack.security.transport.ssl.truststore.path: "certs/elastic-certificates.p12"

xpack.security.http.ssl.enabled: true
#xpack.security.http.ssl.keystore.path: "certs/elastic-certificates.p12"
#xpack.security.http.ssl.truststore.path: "certs/elastic-certificates.p12"
xpack.security.http.ssl.certificate: certs/master-node/fullchain.crt
xpack.security.http.ssl.key: certs/master-node/privkey.key
#xpack.security.http.ssl.certificate_authorities: certs/ca/ca.crt
#

```

As I said, we created another server with the **same elasticsearch version (7.6.2)**

```auto
cluster.name: goulue
node.name: ico-elastic-node-2
node.data: true
node.master: false
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
discovery.zen.ping.unicast.hosts: ["ip_node_1"]
cluster.initial_master_nodes: ["master-node"]
xpack.security.transport.ssl.enabled: true
#xpack.security.transport.ssl.verification_mode: certificate
#xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.certificate: master-node.crt
xpack.security.transport.ssl.certificate_authorities: ca.crt

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.certificate: fullchain.crt
xpack.security.http.ssl.key: privkey.key

```

I took a look at the logs and here is what I found:  
Node 2

```auto
[ico-elastic-node-2] master not discovered yet: have discovered [{node-2}{G5SfrEv0RxaxmYf8urIFtQ}{XpfNNouCQx2HmYfw2AvoQw}{ip_server_2}{ip_server_29300}{dil}{ml.machine_memory=12558602240, xpack.installed=true, ml.max_open_jobs=20}]; discovery will continue using [ip_server_1, 127.0.0.1:9300] from hosts providers and [] from last-known cluster state; node term 0, last-accepted version 0 in term 0`

```

And the master:

```auto
    [master-node] exception caught on transport layer [Netty4TcpChannel{localAddress=/ip:9300, remoteAddress=/ip:36660}], closing connection
    io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: No available authentication scheme

```

Any suggestion? I'm desperate here because our master is yellow and we need to fix this ASAP.

The most important thing for us is to get the master node to green again and with enough space to make it work properly. If there is anything else we can do to avoid data lost without creating the cluster, please let me know.

Thank you for your time.

---

<div class="post-metadata">

**Author:** ![ikakavas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ikakavas/32/34430_2.png) [@ikakavas](https://discuss.elastic.co/u/ikakavas)\
**Post date:** [July 12, 2022, 3:00pm UTC](https://discuss.elastic.co/t/cant-connect-node-to-cluster-on-another-server/309432/2 "2022-07-12T15:00:40Z")

</div>

If you need urgent help, please reach out to your support engineer. This forum is best effort and there is no way to get dedicated help by the community here.

> [@dabit\_coder](#):
>
> Here is the configuration for the main server `/etc/elasticsearch/elasticsearch.yml`

There definitely are more settings in this file, can you share the entire thing ? For instance there is no way to enable TLS ( i.e. `xpack.security.transport.ssl.enabled: true` ) for http and transport layer without actually configuring TLS.

In general, you'd need to configure transport TLS in your new node so that it can join the cluster. You have two options:

- Share the exact configuration with us here and maybe there is a way to let the new node join the cluster ( it _largely_ depends on _how_ TLS is configured already )
- Start the TLS configuration from scratch ( it would probably require cluster downtime ) using our documentation : [Set up basic security for the Elastic Stack | Elasticsearch Guide [8.11] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-basic-setup.html)

---

<div class="post-metadata">

**Author:** ![dabit\_coder](https://avatars.discourse-cdn.com/v4/letter/d/91b2a8/32.png) [@dabit\_coder](https://discuss.elastic.co/u/dabit_coder)\
**Post date:** [July 12, 2022, 3:19pm UTC](https://discuss.elastic.co/t/cant-connect-node-to-cluster-on-another-server/309432/3 "2022-07-12T15:19:28Z")

</div>

Hey ikakavas. Sorry I didnt put the whole file. Here you can find it for the master node

```auto
cluster.name: goulue
node.name: master-node
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
http.max_content_length: 100mb
discovery.seed_hosts: ["95.179.139.6", "127.0.0.1", "95.179.154.158"]
xpack.license.self_generated.type: "basic"
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
#xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.certificate: "certs/master-node/master-node.crt"
xpack.security.transport.ssl.certificate_authorities: "certs/ca/ca.crt"
#xpack.security.transport.ssl.truststore.path: "certs/elastic-certificates.p12"

xpack.security.http.ssl.enabled: true
#xpack.security.http.ssl.keystore.path: "certs/elastic-certificates.p12"
#xpack.security.http.ssl.truststore.path: "certs/elastic-certificates.p12"
xpack.security.http.ssl.certificate: certs/master-node/fullchain.crt
xpack.security.http.ssl.key: certs/master-node/privkey.key
#xpack.security.http.ssl.certificate_authorities: certs/ca/ca.crt
#

```

here is the whole config for the node 2

```auto
cluster.name: goulue
node.name: ico-elastic-node-2
node.data: true
node.master: false
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: 0.0.0.0
discovery.zen.ping.unicast.hosts: ["95.179.140.41"]
cluster.initial_master_nodes: ["master-node"]
xpack.security.transport.ssl.enabled: true
#xpack.security.transport.ssl.verification_mode: certificate
#xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.certificate: master-node.crt
xpack.security.transport.ssl.certificate_authorities: ca.crt

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.certificate: fullchain.crt
xpack.security.http.ssl.key: privkey.key

```

---

<div class="post-metadata">

**Author:** ![Justin\_Cranford](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/justin_cranford/32/85302_2.png) [@Justin\_Cranford](https://discuss.elastic.co/u/Justin_Cranford)\
**Post date:** [July 12, 2022, 4:16pm UTC](https://discuss.elastic.co/t/cant-connect-node-to-cluster-on-another-server/309432/4 "2022-07-12T16:16:46Z")

</div>

> The developer who created all this configuration is gone and we don't have a way to reach him. We don't have any password for the certificates or anything, in case that is needed.

If you are able to run `elasticsearch-keystore show` on your existing nodes, you can recover the passwords.

The secure setting names depend if you are using keystores like PKCS12 or JKS, or if you are using keys+crts in PEM files. From your full config, it looks like you are using keystores, so the first two commands would apply.

If using keystores:

```auto
bin/elasticsearch-keystore show xpack.security.http.ssl.keystore.secure_password
bin/elasticsearch-keystore show xpack.security.transport.ssl.keystore.secure_password

```

If using key+crt in PEM files:

```auto
bin/elasticsearch-keystore show xpack.security.http.ssl.secure_key_passphrase
bin/elasticsearch-keystore show xpack.security.transport.ssl.secure_key_passphrase

```

---

<div class="post-metadata">

**Author:** ![dabit\_coder](https://avatars.discourse-cdn.com/v4/letter/d/91b2a8/32.png) [@dabit\_coder](https://discuss.elastic.co/u/dabit_coder)\
**Post date:** [July 12, 2022, 4:36pm UTC](https://discuss.elastic.co/t/cant-connect-node-to-cluster-on-another-server/309432/5 "2022-07-12T16:36:32Z")

</div>

It looks like the command show doesnt exist for this

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/8/4811a6eb576db00aea100ccdcb7ded1e7175aa44.png)

---

<div class="post-metadata">

**Author:** ![Justin\_Cranford](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/justin_cranford/32/85302_2.png) [@Justin\_Cranford](https://discuss.elastic.co/u/Justin_Cranford)\
**Post date:** [July 12, 2022, 5:16pm UTC](https://discuss.elastic.co/t/cant-connect-node-to-cluster-on-another-server/309432/6 "2022-07-12T17:16:30Z")

</div>

Sorry about that, show password was added to elasticsearch-keystore in 7.16.0. I see you are on 7.6.2.

> **[Elasticsearch version 7.16.0 | Elasticsearch Guide \[7.17\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/release-notes-7.16.0.html)**

> Infra/Settings
> 
> - Add `show` command to the keystore CLI [#76693](https://github.com/elastic/elasticsearch/pull/76693) (issue: [#57261](https://github.com/elastic/elasticsearch/issues/57261))

> **[elasticsearch-keystore | Elasticsearch Guide \[7.16\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/7.16/elasticsearch-keystore.html)**

---

<div class="post-metadata">

**Author:** ![Justin\_Cranford](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/justin_cranford/32/85302_2.png) [@Justin\_Cranford](https://discuss.elastic.co/u/Justin_Cranford)\
**Post date:** [July 12, 2022, 6:16pm UTC](https://discuss.elastic.co/t/cant-connect-node-to-cluster-on-another-server/309432/7 "2022-07-12T18:16:02Z")

</div>

I did an experiment with 7.6.2 and 7.17.4 to confirm I can use 7.17.4 to show password.

I downloaded both zip archives on Linux, ran create and add in 7.6.2, copied `elasticsearch.keystore` to the 7.17.4 config directory, and successfully ran show to recover a password. You may be able to use this workaround to recover your passwords in elasticsearch 7.6.2.

```auto
cd ~/Downloads
tar xfz elasticsearch-7.6.2-linux-x86_64.tar.gz
tar xfz elasticsearch-7.17.4-linux-x86_64.tar.gz
cd ~/Downloads/elasticsearch-7.6.2
bin/elasticsearch-keystore create
bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password
cp ~/Downloads/elasticsearch-7.6.2/config/elasticsearch.keystore ~/Downloads/elasticsearch-7.17.4/config/elasticsearch.keystore
cd ~/Downloads/elasticsearch-7.17.4
bin/elasticsearch-keystore show xpack.security.http.ssl.keystore.secure_password
password1

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 12, 2022, 6:16pm UTC](https://discuss.elastic.co/t/cant-connect-node-to-cluster-on-another-server/309432/8 "2022-07-12T18:16:02Z")

</div>

elasticsearch 7.6 is [EOL](https://www.elastic.co/support/eol) and no longer supported. Please upgrade ASAP.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

---

<div class="post-metadata">

**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\
**Post date:** [July 13, 2022, 12:48am UTC](https://discuss.elastic.co/t/cant-connect-node-to-cluster-on-another-server/309432/9 "2022-07-13T00:48:45Z")

</div>

Can you confirm that you previously had exactly 1 node, and now you want to add a second node?

If you only had 1 node before, then it's entirely possible that the `xpack.security.transport.ssl.*` configuration _never worked_ because a single node cluster doesn't ever need to make connections between nodes.  
That means, for example, that it's possible that `master-node.crt` isn't signed by `ca.crt`. If that's true, then trying to replicate that setup into a new node is going to fail.

However, the good thing if you currently have just one node is that you can start again with transport SSL, and nothing will break (because you don't use it today).

This guide walks you through setting up TLS for internode communication

- [Set up basic security for the Elastic Stack | Elasticsearch Guide [7.17] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/security-basic-setup.html)

It should be a matter of:

1. Create a new CA (`bin/elasticsearch-certutil ca`)
2. Create a new certificate+key for your nodes (`bin/elasticsearch-certutil cert`)
3. Configure each of your nodes with

```auto
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate 
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.keystore.path: elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: elastic-certificates.p12

```

However, I do recommend that you read the guide (linked above) to understand each of those steps.

---

<div class="post-metadata">

**Author:** ![dabit\_coder](https://avatars.discourse-cdn.com/v4/letter/d/91b2a8/32.png) [@dabit\_coder](https://discuss.elastic.co/u/dabit_coder)\
**Post date:** [July 14, 2022, 1:34pm UTC](https://discuss.elastic.co/t/cant-connect-node-to-cluster-on-another-server/309432/10 "2022-07-14T13:34:30Z")

</div>

I can confirm that we didn't have any node previously.

After some tweaking, I discovered that after disabling the security on the master, the node is still unable to join the cluster, so something else should be happening here.

This is the error I get on the node-2

```auto
 master not discovered yet: have discovered [{ico-elastic-node-2}{G5SfrEv0RxaxmYf8urIFtQ}{VNB1gQ_WTDyQnDxNvC2Y0A}{95.179.154.158}{95.179.154.158:9300}{dil}{ml.machine_memory=12558602240, xpack.installed=true, ml.max_open_jobs=20}]; discovery will continue using [95.179.140.41:9200] from hosts providers and [] from last-known cluster state; node term 74, last-accepted version 0 in term 0

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 11, 2022, 1:35pm UTC](https://discuss.elastic.co/t/cant-connect-node-to-cluster-on-another-server/309432/11 "2022-08-11T13:35:03Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.