Certificate problems adding node to cluster

Elastic Stack Elasticsearch
1

I am preparing to turn on "security" on my production cluster by setting up a test cluster first. I have a single node running fine with security enabled but when I try and bring up a second node with the same configuration it fails to join. On the master we see this in the logs:

[2021-12-21T11:44:48,803][WARN ][o.e.c.s.DiagnosticTrustManager] [secmonprd08] failed to establish trust with server at [130.216.5.128]; the server provided a certificate with subject name [CN=secmonprd01.its.auckland.ac.nz] and fingerprint [6cc5ea53083ea2e04355b5f3d28433cf263a6efd]; the certificate has subject alternative names [DNS:es2.test.insec.auckland.ac.nz,DNS:secmonprd01.its.auckland.ac.nz]; the certificate is issued by [CN=R3,O=Let's Encrypt,C=US]; the certificate is signed by (subject [CN=R3,O=Let's Encrypt,C=US] fingerprint [a053375bfe84e8b748782c7cee15827a6af5a405] {trusted issuer}) signed by (subject [CN=ISRG Root X1,O=Internet Security Research Group,C=US] fingerprint [933c6ddee95c9c41a40f9f50493d82be03ad87bf] {trusted issuer}) which is issued by [CN=DST Root CA X3,O=Digital Signature Trust Co.] (but that issuer certificate was not provided in the chain); the issuing certificate with fingerprint [dac9024f54d8f6df94935fb1732638ca6ad77c13] is trusted in this ssl context ([xpack.security.transport.ssl])
java.security.cert.CertificateException: No subject alternative names matching IP address 130.216.5.128 found

in the config on the new node we have:

xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.http.ssl.key: /etc/elasticsearch/ssl/privkey.pem
xpack.security.http.ssl.certificate: /etc/elasticsearch/ssl/fullchain.pem
xpack.security.transport.ssl.key: /etc/elasticsearch/ssl/privkey.pem
xpack.security.transport.ssl.certificate: /etc/elasticsearch/ssl/fullchain.pem

In particular note that the certificate is a Letsencrypt full.chain which should have all the intermediate certs.

Any idea what is actually wrong?

2

By default, a cluster uses IP addresses (rather than DNS names) to establish connections between nodes.
Your certificate is valid for the DNS names es2.test.insec.auckland.ac.nz and secmonprd01.its.auckland.ac.nz but not for the IP address 130.216.5.128.
The simplest option is to switch to certificate mode verification which will skip the hostname check:

xpack.security.transport.ssl.verification_mode: certificate

We strongly recommend using a dedicated CA for TLS between nodes, rather than a public CA.
See:

2 Likes
3

thanks Tim, changing the config works fine!

I guess I could include the IP as an ASN on the cert as well.

As a security engineer I am curious about the reasoning behind the recommendation not to use certs from a public CA. Is it just because it isn't necessary and there is no need to pay one of the big money machines if you don't need to?

We are a large university and now use Letsencrypt for nearly all our certificates (and we have many hundreds). Previously we had a flat fee subscription that allowed an unlimited number of certificates and we made the decision to us "proper" certificates everywhere (in particular in dev, test and prod environments) as well as backend situations like this.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.