Falha na Configuração em Cluster SSL

Pessoal, estou tentando configurar o cluster em SSL utilizando certificado wildcard da godaddy porém está apresentando erro client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/10.20.10.120:9300, remoteAddress=/10.20.10.123:54850}

Fiz o teste no modo standalone e funcionou normalmente, porém ao realizar configuração para um cluster de 3 nós estou recebendo vários warning's nos logs e o cluster não é formado.

Configuração do node1:

cluster.name: elkhomolog
node.name: node-1
network.host: 10.20.10.120
http.port: 9200
discovery.seed_hosts: ["node1.meudominio.com.br"]
cluster.initial_master_nodes: ["node-1"]
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate 
xpack.security.transport.ssl.key: cert/my_key.key
xpack.security.transport.ssl.certificate: cert/mycert_cert.cer
xpack.security.transport.ssl.certificate_authorities: [ "/etc/elasticsearch/cert/gdroot-g2.crt" ]
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key:  cert/my_key.key 
xpack.security.http.ssl.certificate: cert/mycert_cert.cer
xpack.security.http.ssl.certificate_authorities: [ "/etc/elasticsearch/cert/gdroot-g2.crt" ]

Configuração do node 2 e 3:

cluster.name: elkhomolog
node.name: node-2
network.host: 10.20.10.122
http.port: 9200
discovery.seed_hosts: ["node1.meudominio.com.br"]
cluster.initial_master_nodes: ["node-1"]
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate 
xpack.security.transport.ssl.key: cert/my_key.key
xpack.security.transport.ssl.certificate: cert/mycert_cert.cer
xpack.security.transport.ssl.certificate_authorities: [ "/etc/elasticsearch/cert/gdroot-g2.crt" ]
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key:  cert/my_key.key 
xpack.security.http.ssl.certificate: cert/mycert_cert.cer
xpack.security.http.ssl.certificate_authorities: [ "/etc/elasticsearch/cert/gdroot-g2.crt" ]

Log recebido no node1:

[WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [node-1] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/10.20.10.120:9300, remoteAddress=/10.20.10.122:57287}
[WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [node-1] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/10.20.10.120:9300, remoteAddress=/10.20.10.123:54838}

Log recebido nos nodes 2 e 3:

[WARN ][o.e.t.TcpTransport ] [node-2] exception caught on transport layer [Netty4TcpChannel{localAddress=/10.20.10.122:57288, remoteAddress=elkhomolog.meudominio/10.20.10.120:9300}], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[netty-codec-4.1.45.Final.jar:4.1.45.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.45.Final.jar:4.1.45.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.45.Final.jar:4.1.45.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.45.Final.jar:4.1.45.Final]
at java.lang.Thread.run(Thread.java:832) [?:?]
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:325) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:268) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:263) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1340) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1215) ~[?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1158) ~[?:?]
at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]
at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:445) ~[?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1260) ~[?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1247) ~[?:?]
at java.security.AccessController.doPrivileged(AccessController.java:691) ~[?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1192) ~[?:?]

Questão relacionada: TLS between Elasticsearch cluster nodes with Let's Encrypt

Acho que lembrei como resolvi:

  1. Tive que regerar os certificados através do certbot informando o FQDN de cada host
  2. Copiei o fullchain gerado pelo certbot para dentro do diretório de certificados:
cp /etc/letsencrypt/live/host01.meudominio.com.br/fullchain1.pem /u/elastic/elasticsearch/config/certs/
  1. Alterei minha configuração para:
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.key: /u/elastic/elasticsearch/config/certs/host01.meudominio.com.br/privkey1.pem
xpack.security.transport.ssl.certificate: /u/elastic/elasticsearch/config/certs/host01.meudominio.com.br/fullchain1.pem

A partir daí o Elasticsearch passou a confiar pois teve acesso a toda a cadeia do certificado...

Não sou especialista em TLS, foi sofrido mesmo pra conseguir fazer funcionar.