ES 6.5.1 Two nodes not connecting into cluster using ssl certificate

Elastic Stack Elasticsearch
1

I am trying to configure 2 node cluster for Elasticsearch 6.5.1 on Debian 10 machines.
For each cluster I installed Elasticsearch instance from apt-get.
I also generated certificate keys using ca.crt and ca.key files I obtained earlier and the following command:
sudo /usr/share/elasticsearch/bin/elasticsearch-certgen --days 3650 --cert /etc/elasticsearch/certs/ca.crt --key /etc/elasticsearch/certs/ca.key --pass 'my_pass' --in /etc/elasticsearch/certs/certgen.yml --out /etc/elasticsearch/certs/certs_nodes.zip

I unzipped created certificates.
This is elasticsearch.yml content:

cluster.name: my-cluster
node.name: node0
path.data: /var/lib/elasticsearch/data
path.logs: /elasticsearch/logs
bootstrap.memory_lock: true
network.host: [_local_,_ens4_]
indices.queries.cache.size: 30%
http.max_content_length: 500mb
discovery.zen.ping.unicast.hosts: ["192.168.0.2", "192.168.0.3"]
xpack.ssl.key: /etc/elasticsearch/certs/node0/node0.key
xpack.ssl.certificate: /etc/elasticsearch/certs/node0/node0.crt
xpack.ssl.certificate_authorities: [ "/etc/elasticsearch/certs/ca.crt" ]
xpack.security.transport.ssl.enabled: true
xpack.security.http.ssl.enabled: true

Nodes cannot connect with each other to create one cluster.
There is a network connection between machines - I can successfully run ping from one machine to another using ip addresses from elasticsearch.yml configuration.
Sniplet of logs from node0:

[2020-07-21T00:00:02,308][DEBUG][o.e.a.a.c.n.s.TransportNodesStatsAction] [node0] failed to execute on node [HYk3iaRASTevMFIrV4Op4g]
org.elasticsearch.transport.RemoteTransportException: [node1][192.168.0.3:9300][cluster:monitor/nodes/stats[n]]
Caused by: org.elasticsearch.ElasticsearchSecurityException: missing authentication token for action [cluster:monitor/nodes/stats[n]]
        (...)
[2020-07-21T00:00:02,313][INFO ][o.e.c.s.MasterService    ] [node0] zen-disco-node-failed({node1}{HYk3iaRASTevMFIrV4Op4g}{8NxhdGVfTemrzxcgOqg4KA}{192.168.0.3}{192.168.0.3:9300}{ml.machine_memory=7839653888, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true}), reason(failed to ping, tried [3] times, each with maximum [30s] timeout)[{node1}{HYk3iaRASTevMFIrV4Op4g}{8NxhdGVfTemrzxcgOqg4KA}{192.168.0.3}{192.168.0.3:9300}{ml.machine_memory=7839653888, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true} failed to ping, tried [3] times, each with maximum [30s] timeout], reason: removed {{node1}{HYk3iaRASTevMFIrV4Op4g}{8NxhdGVfTemrzxcgOqg4KA}{192.168.0.3}{192.168.0.3:9300}{ml.machine_memory=7839653888, ml.max_open_jobs=20, xpack.installed=true, ml.enabled=true},}

Sniplet of logs from node1:

[2020-07-20T08:47:35,384][WARN ][o.e.x.s.t.n.SecurityNetty4ServerTransport] [node1] send message failed [channel: NettyTcpChannel{localAddress=/192.168.0.3:9300, remoteAddress=/192.168.0.2:44828}]
javax.net.ssl.SSLException: SSLEngine closed already
        at io.netty.handler.ssl.SslHandler.wrap(...)(Unknown Source) ~[?:?]

What I am missing or doing wrong? How to configure this nodes to allow communication between them?

2

Do you have connectivity between the nodes? Do you have a valid license (security was not made part of the basic license until version 6.8)?

Any other clues in the logs?

4

You protected your ca with a password. Have you configured the keystore and truststore to use this password ?

5

I have invoked the following command:
printf "my_pass" | sudo /usr/share/elasticsearch/bin/elasticsearch-keystore add -x xpack.security.transport.ssl.secure_key_passphrase

Should I do something more?

6

(You need to check if the following will work with your current version of ES)

If the signed certificate is in PKCS#12 format, use the following commands:

bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password
bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password

If the certificate is in PEM format, use the following commands:

bin/elasticsearch-keystore add xpack.security.transport.ssl.secure_key_passphrase

You can check this link for more details
https://www.elastic.co/guide/en/elasticsearch/reference/master/configuring-tls.html

7

@Christian_Dahlqvist Thank you for hinting license. That was the blocker.
After putting xpack license whole cluster started to communicate.