ES 6.5.1 Two nodes not connecting into cluster using ssl certificate

I am trying to configure 2 node cluster for Elasticsearch 6.5.1 on Debian 10 machines.
For each cluster I installed Elasticsearch instance from apt-get.
I also generated certificate keys using ca.crt and ca.key files I obtained earlier and the following command:
sudo /usr/share/elasticsearch/bin/elasticsearch-certgen --days 3650 --cert /etc/elasticsearch/certs/ca.crt --key /etc/elasticsearch/certs/ca.key --pass 'my_pass' --in /etc/elasticsearch/certs/certgen.yml --out /etc/elasticsearch/certs/certs_nodes.zip

I unzipped created certificates.
This is elasticsearch.yml content:

cluster.name: my-cluster
node.name: node0
path.data: /var/lib/elasticsearch/data
path.logs: /elasticsearch/logs
bootstrap.memory_lock: true
network.host: [_local_,_ens4_]
indices.queries.cache.size: 30%
http.max_content_length: 500mb
discovery.zen.ping.unicast.hosts: ["192.168.0.2", "192.168.0.3"]
xpack.ssl.key: /etc/elasticsearch/certs/node0/node0.key
xpack.ssl.certificate: /etc/elasticsearch/certs/node0/node0.crt
xpack.ssl.certificate_authorities: [ "/etc/elasticsearch/certs/ca.crt" ]
xpack.security.transport.ssl.enabled: true
xpack.security.http.ssl.enabled: true

Nodes cannot connect with each other to create one cluster.
There is a network connection between machines - I can successfully run ping from one machine to another using ip addresses from elasticsearch.yml configuration.
Sniplet of logs from node0:

[2020-07-21T00:00:02,308][DEBUG][o.e.a.a.c.n.s.TransportNodesStatsAction] [node0] failed to execute on node [HYk3iaRASTevMFIrV4Op4g]
org.elasticsearch.transport.RemoteTransportException: [node1][192.168.0.3:9300][cluster:monitor/nodes/stats[n]]
Caused by: org.elasticsearch.ElasticsearchSecurityException: missing authentication token for action [cluster:monitor/nodes/stats[n]]
        (...)
[2020-07-21T00:00:02,313][INFO ][o.e.c.s.MasterService    ] [node0] zen-disco-node-failed({node1}{HYk3iaRASTevMFIrV4Op4g}{8NxhdGVfTemrzxcgOqg4KA}{192.168.0.3}{192.168.0.3:9300}{ml.machine_memory=7839653888, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}), reason(failed to ping, tried [3] times, each with maximum [30s] timeout)[{node1}{HYk3iaRASTevMFIrV4Op4g}{8NxhdGVfTemrzxcgOqg4KA}{192.168.0.3}{192.168.0.3:9300}{ml.machine_memory=7839653888, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} failed to ping, tried [3] times, each with maximum [30s] timeout], reason: removed {{node1}{HYk3iaRASTevMFIrV4Op4g}{8NxhdGVfTemrzxcgOqg4KA}{192.168.0.3}{192.168.0.3:9300}{ml.machine_memory=7839653888, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true},}

Sniplet of logs from node1:

[2020-07-20T08:47:35,384][WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [node1] send message failed [channel: NettyTcpChannel{localAddress=/192.168.0.3:9300, remoteAddress=/192.168.0.2:44828}]
javax.net.ssl.SSLException: SSLEngine closed already
        at io.netty.handler.ssl.SslHandler.wrap(...)(Unknown Source) ~[?:?]

What I am missing or doing wrong? How to configure this nodes to allow communication between them?

Do you have connectivity between the nodes? Do you have a valid license (security was not made part of the basic license until version 6.8)?

Any other clues in the logs?

You protected your ca with a password. Have you configured the keystore and truststore to use this password ?

I have invoked the following command:
printf "my_pass" | sudo /usr/share/elasticsearch/bin/elasticsearch-keystore add -x xpack.security.transport.ssl.secure_key_passphrase

Should I do something more?

(You need to check if the following will work with your current version of ES)

If the signed certificate is in PKCS#12 format, use the following commands:

bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password

If the certificate is in PEM format, use the following commands:

bin/elasticsearch-keystore add xpack.security.transport.ssl.secure_key_passphrase

You can check this link for more details
https://www.elastic.co/guide/en/elasticsearch/reference/master/configuring-tls.html

@Christian_Dahlqvist Thank you for hinting license. That was the blocker.
After putting xpack license whole cluster started to communicate.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.