# TLS between Elasticsearch cluster nodes with Let's Encrypt

**URL:** <https://discuss.elastic.co/t/tls-between-elasticsearch-cluster-nodes-with-lets-encrypt/229559>\
**Category:** Elasticsearch\
**Created:** [April 24, 2020, 12:01am UTC](https://discuss.elastic.co/t/tls-between-elasticsearch-cluster-nodes-with-lets-encrypt/229559 "2020-04-24T00:01:39Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![ffknob](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ffknob/32/37008_2.png) [@ffknob](https://discuss.elastic.co/u/ffknob)\
**Post date:** [April 24, 2020, 12:01am UTC](https://discuss.elastic.co/t/tls-between-elasticsearch-cluster-nodes-with-lets-encrypt/229559/1 "2020-04-24T00:01:40Z")

</div>

Hello,

I need some help setting a cluster with N nodes and TLS using Let's Encrypt. I've already configured clusters using self-signed (`elasticsearch-certutil`) certificates, but I really don't know much about PKI infrasctructure, so I'll appreciate any help...

1. After generating certificates with certbot I copyied them from `/etc/letsencrypt/archive/host01.mydomain.com` to `/u/elastic/elasticsearch/config/certs/host01.mydomain.com`

2. I then changed ownership and permissions, like this

```auto
sudo chmod 750 /u/elastic/elasticsearch/config/certs/
sudo chmod 640 /u/elastic/elasticsearch/config/certs/host01.mydomain.com/cert1.pem /u/elastic/elasticsearch/config/certs/host01.mydomain.com/chain1.pem /u/elastic/elasticsearch/config/certs/host01.mydomain.com/fullchain1.pem /u/elastic/elasticsearch/config/certs/host01.mydomain.com/privkey1.pem
sudo chown -R elastic:elastic /u/elastic/elasticsearch/config/certs

```

1. Here is what one of my `elasticsearch.yml` looks like

```auto
cluster.name: supercluster
node.name: node01
network.host: _eth0_
discovery.seed_hosts: node02,node03
cluster.initial_master_nodes: host01,hos02,hos03
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.key: /u/elastic/elasticsearch/config/certs/host01.mydomain.com/privkey1.pem
xpack.security.transport.ssl.certificate: /u/elastic/elasticsearch/config/certs/host01.mydomain.com/cert1.pem
xpack.security.transport.ssl.certificate_authorities: ["/u/elastic/elasticsearch/config/certs/host01.mydomain.com/fullchain1.pem"]
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: /u/elastic/elasticsearch/config/certs/host01.mydomain.com/privkey1.pem
xpack.security.http.ssl.certificate: /u/elastic/elasticsearch/config/certs/host01.mydomain.com/cert1.pem
xpack.security.http.ssl.certificate_authorities: ["/u/elastic/elasticsearch/config/certs/host01.mydomain.com/fullchain1.pem"]

```

1. when I start `node01`on `host01`, everything works

2. But when I start `node02` on `host02`, I get this on `node01`:

```auto
[2020-04-23T20:44:35,393][WARN][o.e.x.c.s.t.n.SecurityNetty4Transport] [node01] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/999.999.999.999:9300, remoteAddress=/999.999.999.999:43198}

```

And this on `node02`:

```auto
[2020-04-23T20:44:35,363][WARN][o.e.c.s.DiagnosticTrustManager] [node02] failed to establish trust with server at [host01]; the server provided a certificate with subject name [CN=host01.mydomain.com] and fingerprint [...]; the certificate has subject alternative names [DNS:host01.mydomain.com]; the certificate is issued by [CN=Let's Encrypt Authority X3,O=Let's Encrypt,C=US]; the certificate is signed by (subject [CN=Let's Encrypt Authority X3,O=Let's Encrypt,C=US] fingerprint [...] {trusted issuer}) which is issued by [CN=DST Root CA X3,O=Digital Signature Trust Co.] (but that issuer certificate was not provided in the chain); this ssl context ([xpack.security.transport.ssl]) is not configured to trust that issuer
java.security.cert.CertificateException: No subject alternative DNS name matching host01 found.
	at sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:211) ~[?:?]
	at sun.security.util.HostnameChecker.match(HostnameChecker.java:102) ~[?:?]
	at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:452) ~[?:?]
	at sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:426) ~[?:?]
	at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:292) ~[?:?]
	at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144) ~[?:?]
	at org.elasticsearch.common.ssl.DiagnosticTrustManager.checkServerTrusted(DiagnosticTrustManager.java:110) [elasticsearch-ssl-config-7.6.2.jar:7.6.2]
	at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1317) [?:?]
	at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1214) [?:?]
	at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1157) [?:?]
	at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) [?:?]
	at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) [?:?]
	at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1260) [?:?]
	at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1247) [?:?]
	at java.security.AccessController.doPrivileged(AccessController.java:691) [?:?]
	at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1192) [?:?]
	at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1502) [netty-handler-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1516) [netty-handler-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1400) [netty-handler-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1227) [netty-handler-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1274) [netty-handler-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:503) [netty-codec-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:442) [netty-codec-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:281) [netty-codec-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:600) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:554) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1050) [netty-common-4.1.43.Final.jar:4.1.43.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.43.Final.jar:4.1.43.Final]
	at java.lang.Thread.run(Thread.java:830) [?:?]

```

So, my questions are:

1. How do I make the nodes trust `Let's Encrypt`?

2. I copied the certificate files from `certbot`'s directory to Elasticsearch's config directory. Read somewhere this is a Java secutiry requirement. But what will happen when Certbot updates the certificate?

Thank you for taking the time to help me figure this on out

---

<div class="post-metadata">

**Author:** ![Luca\_Belluccini](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/luca_belluccini/32/33239_2.png) [@Luca\_Belluccini](https://discuss.elastic.co/u/Luca_Belluccini)\
**Post date:** [April 24, 2020, 12:28am UTC](https://discuss.elastic.co/t/tls-between-elasticsearch-cluster-nodes-with-lets-encrypt/229559/2 "2020-04-24T00:28:21Z")

</div>

We have a really [old blog post](https://www.elastic.co/blog/x-pack-security-for-elasticsearch-with-lets-encrypt-certificates) regarding Let's Encrypt but a part of the procedure is similar.

> [@ffknob](#):
>
> No subject alternative DNS name matching host01 found.

The nodes do not trust each other because the certificate you've generated is probably valid for `host01.mydomain.com`, but not for `host01`.  
The TLS validation includes checking the DNS in the certificate with the one provided in the configuration.

One approach might be to deactivate the DNS/Hostname verification, adding:

```auto
xpack.security.transport.ssl.verification_mode: certificate

```

Please consider the security implications of this approach.

> [@ffknob](#):
>
> I copied the certificate files from `certbot` 's directory to Elasticsearch's config directory. Read somewhere this is a Java secutiry requirement. But what will happen when Certbot updates the certificate?

You would need to overwrite the existing files in the Elasticsearch configuration directory.  
From our [doc](https://www.elastic.co/guide/en/elasticsearch/reference/current/configuring-tls.html):

> Elasticsearch monitors all files such as certificates, keys, keystores, or truststores that are configured as values of TLS-related node settings. If you update any of these files (for example, when your hostnames change or your certificates are due to expire), Elasticsearch reloads them. The files are polled for changes at a frequency determined by the global Elasticsearch `resource.reload.interval.high` setting, which defaults to 5 seconds.

---

<div class="post-metadata">

**Author:** ![ffknob](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ffknob/32/37008_2.png) [@ffknob](https://discuss.elastic.co/u/ffknob)\
**Post date:** [April 24, 2020, 1:14am UTC](https://discuss.elastic.co/t/tls-between-elasticsearch-cluster-nodes-with-lets-encrypt/229559/3 "2020-04-24T01:14:06Z")

</div>

Hi @Luca_Belluccini,

first of all, thank you for taking the time to answer my question.

I certainly now that blog, spent the last few hours on it.

You are right, I should've set the hostnames to their FQDN, so`host01.mydomain.com`. Just did it, but it kept complaining that it doesn't trust in the certificate's issuer (Let's Encrypt).

Maybe it is a mather of including some root certificate in the `xpack.security.transport.ssl.certificate_authorities` parameter, or making CentOS trust Let's Encrypt root certificate... I'm really not experienced with PKI infrasctructure. Actually in that blog there is something in that matter:

> If your Java cacerts keystore does not contain the [DST Root CA X3](https://www.identrust.com/certificates/trustid/root-download-x3.html) certificate or newer [ISRG Root X1](https://letsencrypt.org/certificates/) CA certificate for any reason, you could also provide the Certificate Authorities certificates directly to Elasticsearch via the following configuration. This was not required with an updated version of CentOS 6, but you may find that either the DST Root CA X3 certificate or the newer ISRG root CA used by Let's Encrypt may not be recognized by some older OS or web browser versions:

That means I have to download a root certificate from LEt's Encrypt and tell Elasticsearch to consider that ?

One last thing, about that second question, nice to know that Elasticsearch keeps on the files to see if they change. But the thing is, since I had to copy the certificates from the official directory to inside Elasticsearch's `config` directory, whenecer certbot updates my certificates (3 months from now) I'll have to have something that executes thi copy again... right?

Or is that something that changed from the time the blog was written and now I can point to certificates outside the `config` dir?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 22, 2020, 1:15am UTC](https://discuss.elastic.co/t/tls-between-elasticsearch-cluster-nodes-with-lets-encrypt/229559/4 "2020-05-22T01:15:34Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.