TLS between Elasticsearch cluster nodes with Let's Encrypt

ffknob · April 24, 2020, 12:01am

Hello,

I need some help setting a cluster with N nodes and TLS using Let's Encrypt. I've already configured clusters using self-signed (elasticsearch-certutil) certificates, but I really don't know much about PKI infrasctructure, so I'll appreciate any help...

After generating certificates with certbot I copyied them from /etc/letsencrypt/archive/host01.mydomain.com to /u/elastic/elasticsearch/config/certs/host01.mydomain.com
I then changed ownership and permissions, like this

sudo chmod 750 /u/elastic/elasticsearch/config/certs/
sudo chmod 640 /u/elastic/elasticsearch/config/certs/host01.mydomain.com/cert1.pem /u/elastic/elasticsearch/config/certs/host01.mydomain.com/chain1.pem /u/elastic/elasticsearch/config/certs/host01.mydomain.com/fullchain1.pem /u/elastic/elasticsearch/config/certs/host01.mydomain.com/privkey1.pem
sudo chown -R elastic:elastic /u/elastic/elasticsearch/config/certs

Here is what one of my elasticsearch.yml looks like

cluster.name: supercluster
node.name: node01
network.host: _eth0_
discovery.seed_hosts: node02,node03
cluster.initial_master_nodes: host01,hos02,hos03
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.key: /u/elastic/elasticsearch/config/certs/host01.mydomain.com/privkey1.pem
xpack.security.transport.ssl.certificate: /u/elastic/elasticsearch/config/certs/host01.mydomain.com/cert1.pem
xpack.security.transport.ssl.certificate_authorities: [ "/u/elastic/elasticsearch/config/certs/host01.mydomain.com/fullchain1.pem" ]
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: /u/elastic/elasticsearch/config/certs/host01.mydomain.com/privkey1.pem
xpack.security.http.ssl.certificate: /u/elastic/elasticsearch/config/certs/host01.mydomain.com/cert1.pem
xpack.security.http.ssl.certificate_authorities: [ "/u/elastic/elasticsearch/config/certs/host01.mydomain.com/fullchain1.pem" ]

when I start node01on host01, everything works
But when I start node02 on host02, I get this on node01:

[2020-04-23T20:44:35,393][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [node01] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/999.999.999.999:9300, remoteAddress=/999.999.999.999:43198}

And this on node02:

[2020-04-23T20:44:35,363][WARN ][o.e.c.s.DiagnosticTrustManager] [node02] failed to establish trust with server at [host01]; the server provided a certificate with subject name [CN=host01.mydomain.com] and fingerprint [...]; the certificate has subject alternative names [DNS:host01.mydomain.com]; the certificate is issued by [CN=Let's Encrypt Authority X3,O=Let's Encrypt,C=US]; the certificate is signed by (subject [CN=Let's Encrypt Authority X3,O=Let's Encrypt,C=US] fingerprint [...] {trusted issuer}) which is issued by [CN=DST Root CA X3,O=Digital Signature Trust Co.] (but that issuer certificate was not provided in the chain); this ssl context ([xpack.security.transport.ssl]) is not configured to trust that issuer
java.security.cert.CertificateException: No subject alternative DNS name matching host01 found.
	at sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:211) ~[?:?]
	at sun.security.util.HostnameChecker.match(HostnameChecker.java:102) ~[?:?]
	at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:452) ~[?:?]
	at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:426) ~[?:?]
	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:292) ~[?:?]
	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144) ~[?:?]
	at org.elasticsearch.common.ssl.DiagnosticTrustManager.checkServerTrusted(DiagnosticTrustManager.java:110) [elasticsearch-ssl-config-7.6.2.jar:7.6.2]
	at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1317) [?:?]
	at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1214) [?:?]
	at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1157) [?:?]
	at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) [?:?]
	at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) [?:?]
	at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1260) [?:?]
	at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1247) [?:?]
	at java.security.AccessController.doPrivileged(AccessController.java:691) [?:?]
	at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1192) [?:?]
	at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1502) [netty-handler-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1516) [netty-handler-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1400) [netty-handler-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227) [netty-handler-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) [netty-handler-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:503) [netty-codec-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:442) [netty-codec-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:281) [netty-codec-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:600) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:554) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050) [netty-common-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.43.Final.jar:4.1.43.Final]
	at java.lang.Thread.run(Thread.java:830) [?:?]

So, my questions are:

How do I make the nodes trust Let's Encrypt?
I copied the certificate files from certbot's directory to Elasticsearch's config directory. Read somewhere this is a Java secutiry requirement. But what will happen when Certbot updates the certificate?

Thank you for taking the time to help me figure this on out

Luca_Belluccini · April 24, 2020, 12:28am

We have a really old blog post regarding Let's Encrypt but a part of the procedure is similar.

The nodes do not trust each other because the certificate you've generated is probably valid for host01.mydomain.com, but not for host01.
The TLS validation includes checking the DNS in the certificate with the one provided in the configuration.

One approach might be to deactivate the DNS/Hostname verification, adding:

xpack.security.transport.ssl.verification_mode: certificate

Please consider the security implications of this approach.

You would need to overwrite the existing files in the Elasticsearch configuration directory.
From our doc:

Elasticsearch monitors all files such as certificates, keys, keystores, or truststores that are configured as values of TLS-related node settings. If you update any of these files (for example, when your hostnames change or your certificates are due to expire), Elasticsearch reloads them. The files are polled for changes at a frequency determined by the global Elasticsearch resource.reload.interval.high setting, which defaults to 5 seconds.

ffknob · April 24, 2020, 1:14am

Hi @Luca_Belluccini,

first of all, thank you for taking the time to answer my question.

I certainly now that blog, spent the last few hours on it.

You are right, I should've set the hostnames to their FQDN, sohost01.mydomain.com. Just did it, but it kept complaining that it doesn't trust in the certificate's issuer (Let's Encrypt).

Maybe it is a mather of including some root certificate in the xpack.security.transport.ssl.certificate_authorities parameter, or making CentOS trust Let's Encrypt root certificate... I'm really not experienced with PKI infrasctructure. Actually in that blog there is something in that matter:

If your Java cacerts keystore does not contain the DST Root CA X3 certificate or newer ISRG Root X1 CA certificate for any reason, you could also provide the Certificate Authorities certificates directly to Elasticsearch via the following configuration. This was not required with an updated version of CentOS 6, but you may find that either the DST Root CA X3 certificate or the newer ISRG root CA used by Let's Encrypt may not be recognized by some older OS or web browser versions:

That means I have to download a root certificate from LEt's Encrypt and tell Elasticsearch to consider that ?

One last thing, about that second question, nice to know that Elasticsearch keeps on the files to see if they change. But the thing is, since I had to copy the certificates from the official directory to inside Elasticsearch's config directory, whenecer certbot updates my certificates (3 months from now) I'll have to have something that executes thi copy again... right?

Or is that something that changed from the time the blog was written and now I can point to certificates outside the config dir?

system · May 22, 2020, 1:15am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.