Add a new Elasticsearch to TLS/SSL cluster


I have a cluster with 2 nodes of Elasticsearch. TLS/SSL is enables according to the bellow blog:

Everything is OK and monitor-node-01 with IP and monitor-node-02 with IP works correctly with TLS/SSL. But now, I want to add a new node to this cluster.

I used /usr/share/elasticsearch/bin/elasticsearch-certutil to create new certificate for the new node and problem not solved:

elasticsearch-certutil cert --name node3 --dns

Certuril gave me ELK03-SOFT.p12 and I extracted public and private key:

openssl pkcs12 -in ELK03-SOFT.p12 -nocerts -out ELK03-SOFT.key
openssl pkcs12 -in ELK03-SOFT.p12 -clcerts -nokeys -out ELK03-SOFT.crt

In monitor-node-03 the elasticsearch.yml is: true true /path/to/node3.key /path/to/node3.crt /path/to/ca.crt true /path/to/node3.key /path/to/node3.crt /path/to/ca.crt

The problem persists, then I used CA of other 2 nodes that created before(according to the above blog) ca.cert and ca.key that was on monitor-node-01 and monitor-node-02, beside, I created a public and private key with bellow ways and singed them with CA

openssl req -new -newkey rsa:2048 -keyout ELK03.key -out ELK03.csr
openssl x509 -req -in ELK03.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out ELK03.crt -days 365

openssl genpkey -algorithm RSA -out ELK03.key -aes256 -pass pass:
openssl genpkey -algorithm RSA -out ELK03.key -aes256

openssl genpkey -algorithm RSA -out ELK03.key -aes256
openssl req -new -key ELK03.key -out ELK03.csr

openssl x509 -in ./cert.pem -noout -modulus | openssl sha1
openssl rsa -in ./privkey.pem -noout -modulus | openssl sha1

The new Elasticsearch is monitor-node-03 with IP This is the log of new Elasticsearch:

[2023-10-18T17:05:49,479][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [monitor-node-03] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/, remoteAddress=ELK01-SOFT/, profile=default}

On the master node monitor-node-1 when ELK03 (the new node) try to connect shows the bellow log:

[2023-10-21T14:09:56,596][WARN ][o.e.t.TcpTransport       ] [monitor-node-01] exception caught on transport layer [Netty4TcpChannel{localAddress=/, remoteAddress=/, profile=default}], closing connection
io.netty.handler.codec.DecoderException: PKIX path validation failed: Path does not chain with any of the trust anchors
Caused by: PKIX path validation failed: Path does not chain with any of the trust anchors

This is my master config:

# ======================== Elasticsearch Configuration =========================
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
# Please consult the documentation for further information on configuration options:
# ---------------------------------- Cluster -----------------------------------
# Use a descriptive name for your cluster:
# Monitoring_Service
# ------------------------------------ Node ------------------------------------
# Use a descriptive name for the node:
# monitor-node-01
  #node.master: true true
node.roles: [ master, data, data_hot, data_cold, ingest, transform, ml, remote_cluster_client ]
# Add custom attributes to the node:
#node.attr.rack: r1
# ----------------------------------- Paths ------------------------------------
# Path to directory where to store the data (separate multiple locations by comma):
# /var/lib/elasticsearch
# Path to log files:
path.logs: /var/log/elasticsearch
# ----------------------------------- Memory -----------------------------------
# Lock the memory on startup:
bootstrap.memory_lock: false
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
# Elasticsearch performs poorly when the system is swapping the memory.
# ---------------------------------- Network -----------------------------------
# By default Elasticsearch is only accessible on localhost. Set a different
# address here to expose this node on the network:
# By default Elasticsearch listens for HTTP traffic on the first free port it
# finds starting at 9200. Set a specific HTTP port here:
http.port: 9200
# For more information, consult the network module documentation.
# --------------------------------- Discovery ----------------------------------
discovery.zen.minimum_master_nodes: 1 ["ELK01-SOFT","ELK02-SOFT","ELK03-SOFT"]
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["", "[::1]"]
#discovery.seed_hosts: ["host1", "host2"]
# Bootstrap the cluster using an initial set of master-eligible nodes:
#cluster.initial_master_nodes: ["node-1", "node-2"]
# For more information, consult the discovery and cluster formation module documentation.
# ---------------------------------- Various -----------------------------------
# Require explicit names when deleting indices:
#action.destructive_requires_name: true
# ---------------------------------- Security ----------------------------------
#                                 *** WARNING ***
# Elasticsearch security features are not enabled by default.
# These features are free, but require configuration changes to enable them.
# This means that users don’t have to provide credentials and can get full access
# to the cluster. Network connections are also not encrypted.
# To protect your data, we strongly encourage you to enable the Elasticsearch security features.
# Refer to the following documentation for instructions.
# true true certs/node1.key certs/node1.crt certs/ca.crt true certs/node1.key certs/node1.crt certs/ca.crt true

The ELK02-SOFT is the second node and new node is ELK03-SOFT

How can I join this node to the cluster?

Hi @Farid_Niasti Welcome to the community.

What Version?

Question did you create the cert for the new node with the CA that you used to create the certs for the first 2 nodes?

This looks like you are missing providing the original CA, then the 3 nodes will not trust each other

--ca <file_path>
Specifies the path to an existing CA key pair (in PKCS#12 format). This parameter is only applicable to the cert parameter.

Of course you could create new CA and certs for all 3 nodes.

1 Like


I'm using 7.16.2
And I did. I created new cert for monitor-node-03 and signed with the CA

But after signing new node's certificate, I got the bellow error:

Caused by: java.lang.IllegalStateException: Error parsing Private Key from: /etc/elasticsearch/certs/ELK03-SOFT.key
Caused by: PKCS#8 Private Key is encrypted with unsupported PBES2 algorithm [1.2.840.113549.3.7]
Caused by: PBE parameter parsing error: expecting the object identifier for AES cipher

After that, I tried another ways...

I think I should remove all certificates, and then create new certificates for all 3 Elasticsearch nodes, also Logstash and Kibana...

Various versions of openssl do weird and inconsistent things across different commands.

It looks like openssl has extracted your private key in PKCS#8 format, and encrypted it with 3DES (1.2.840.113549.3.7), which Elasticsearch does not support.

Generally, you should avoid using openssl to reprocess certificates and keys that came from elasticsearch-certutil.
If you want your certs in PEM format, just pass --pem to certutil
If you want to use openssl, just call openssl instead.
Trying to mix-and-match openssl with certutil tends to lead to these sorts of issues.


This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.