Unable to establish the connection between two clusters

Subrahmanyam_Veerank · March 1, 2024, 5:07am

Hi sir,

im trying to establish the connection between the two clusters (local and remote) and i have used the same CA of local cluster to generate the certificates of the remote cluster node. In kibana remote clusters, i have added the details of ip address:port number under seeds and remaining things as by default only. But im unable to establish the connection with remote cluster and when i checked in the logs of remote cluster node it was getting the certificate trust issue. Is there anything that i was missing?

setup details:

local cluster: 1 master and 3 (data & ingest)
remote cluster: 1 node (having all the roles)

Request:

PUT _cluster/settings
{
"persistent": {
"cluster": {
"remote": {
"remote-cluster": {
"skip_unavailable": false,
"mode": "sniff",
"proxy_address": null,
"proxy_socket_connections": null,
"server_name": null,
"seeds": [
"xx.xxx.xxx.xxx:9300"
],
"node_connections": 3
}
}
}
}
}

rtwolfe94022 · March 1, 2024, 5:12am

To resolve the certificate trust issue between your local and remote Elasticsearch clusters, ensure the following:

Both clusters trust the CA: The CA used to generate the node certificates must be imported into the truststores of both clusters.
Certificates have correct SAN entries: Check that the certificates include the correct Subject Alternative Names (SANs) for IP addresses or DNS names.
SSL is enabled and correctly configured: Verify xpack.security.transport.ssl settings in elasticsearch.yml for both clusters, ensuring paths to certificates and keys are correct.
Firewall and network connectivity: Ensure no network issues or firewalls block the connection on the transport port (default 9300).
Review logs for specific errors: Check Elasticsearch logs for detailed SSL or connection error messages.

Ensure the setup is correctly pointing to the certificate files and that all configurations align with security requirements.

Subrahmanyam_Veerank · March 1, 2024, 5:15am

sir..how to import into the truststore?

Subrahmanyam_Veerank · March 1, 2024, 8:33am

sir... i have generated ca.crt and ca.key on the local cluster previously using below command.

./elasticsearch-certutil ca --pem --out /etc/elasticsearch/certs/ca.zip

now i have copied the ca.crt and ca.key to the remote cluster /etc/elasticsearch/certs . Kindly share the commands to make import into the truststores to establish the trust.

Enable encryption and mutual authentication between cluster nodes

xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
keystore.path: certs/transport.p12
truststore.path: certs/transport.p12

the transport.ssl is by default on both the clusters..

Subrahmanyam_Veerank · March 7, 2024, 4:27am

sir..can you please share the procedure?

Subrahmanyam_Veerank · March 14, 2024, 8:57am

kindly help us in this regard?

Subrahmanyam_Veerank · March 18, 2024, 9:38am

Can anyone help me regarding this issue?

TimV · March 19, 2024, 2:35am

Please show the full log, including the SSL diagnostic log message, and also your most recent configuration.

Subrahmanyam_Veerank · March 19, 2024, 4:49am

Sir please find the logs and SSL related configurations below. Im using the same CA [Self signed] for both the clusters.

Remote cluster settings in kibana [local cluster]:
=================================
PUT _cluster/settings
{
  "persistent": {
    "cluster": {
      "remote": {
        "remote-cluster": {
          "skip_unavailable": false,
          "mode": "sniff",
          "proxy_address": null,
          "proxy_socket_connections": null,
          "server_name": null,
          "seeds": [
            "132.17.64.68:9300"
          ],
          "node_connections": 3
        }
      }
    }
  }
}


Remote cluster logs:
===================

[2024-03-19T10:02:12,802][INFO ][o.e.m.j.JvmGcMonitorService] [node-1] [gc][177] overhead, spent [301ms] collecting in the last [1s]
[2024-03-19T10:03:35,800][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [node-1] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/132.17.64.68:9300, remoteAddress=/122.17.14.63:50516, profile=default}
[2024-03-19T10:03:35,825][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [node-1] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/132.17.64.68:9300, remoteAddress=/122.17.14.63:50520, profile=default}
[2024-03-19T10:03:35,850][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [node-1] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/132.17.64.68:9300, remoteAddress=/122.17.14.64:45184, profile=default}
[2024-03-19T10:03:35,927][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [node-1] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/132.17.64.68:9300, remoteAddress=/122.17.14.63:50518, profile=default}
[2024-03-19T10:03:36,275][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [node-1] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/132.17.64.68:9300, remoteAddress=/122.17.14.70:37968, profile=default}


Local cluster logs:
==================

[2024-03-19T10:03:34,831][INFO ][o.e.c.s.ClusterSettings  ] [master-node1] updating [cluster.remote.remote-cluster.mode] from [SNIFF] to [sniff]
[2024-03-19T10:03:34,832][INFO ][o.e.c.s.ClusterSettings  ] [master-node1] updating [cluster.remote.remote-cluster.seeds] from [[]] to [["132.17.64.68:9300"]]
[2024-03-19T10:03:34,832][INFO ][o.e.c.s.ClusterSettings  ] [master-node1] updating [cluster.remote.remote-cluster.mode] from [SNIFF] to [sniff]
[2024-03-19T10:03:34,833][INFO ][o.e.c.s.ClusterSettings  ] [master-node1] updating [cluster.remote.remote-cluster.seeds] from [[]] to [["132.17.64.68:9300"]]
[2024-03-19T10:03:35,152][WARN ][o.e.c.s.DiagnosticTrustManager] [master-node1] failed to establish trust with server at [<unknown host>]; the server provided a certificate with subject name [CN=remote-masternode1], fingerprint [039575ec662d29bc50bcbeb939bab407f52ef788], no keyUsage and no extendedKeyUsage; the certificate is valid between [2024-02-28T11:35:46Z] and [2123-02-04T11:35:46Z] (current time is [2024-03-19T04:33:35.152130606Z], certificate dates are valid); the session uses cipher suite [TLS_AES_256_GCM_SHA384] and protocol [TLSv1.3]; the certificate does not have any subject alternative names; the certificate is issued by [CN=Elasticsearch security auto-configuration HTTP CA]; the certificate is signed by (subject [CN=Elasticsearch security auto-configuration HTTP CA] fingerprint [88cd0259c526e732dfc3c4a6c7eba8ffd]) which is self-issued; the [CN=Elasticsearch security auto-configuration HTTP CA] certificate is not trusted in this ssl context ([xpack.security.transport.ssl (with trust configuration: StoreTrustConfig{path=certs/transport.p12, password=<non-empty>, type=PKCS12, algorithm=PKIX})]); this ssl context does trust a certificate with subject [CN=Elasticsearch security auto-configuration HTTP CA] but the trusted certificate has fingerprint [577a707106cd8df5c4cdc48c06b19e25]
sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:318) ~[?:?]
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:267) ~[?:?]
        at sun.security.validator.Validator.validate(Validator.java:256) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:284) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144) ~[?:?]
        at org.elasticsearch.common.ssl.DiagnosticTrustManager.checkServerTrusted(DiagnosticTrustManager.java:100) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1296) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1195) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1138) ~[?:?]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:393) ~[?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:476) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1273) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1260) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:714) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1205) ~[?:?]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1559) ~[?:?]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1405) ~[?:?]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246) ~[?:?]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
        at java.lang.Thread.run(Thread.java:1583) ~[?:?]
Caused by: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:157) ~[?:?]
        at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:83) ~[?:?]
        at java.security.cert.CertPathValidator.validate(CertPathValidator.java:309) ~[?:?]
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:313) ~[?:?]
        ... 36 more
[2024-03-19T10:03:35,162][WARN ][o.e.t.TcpTransport       ] [master-node1] exception caught on transport layer [Netty4TcpChannel{localAddress=/122.17.14.70:37968, remoteAddress=/132.17.64.68:9300, profile=default}], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:499) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
        at java.lang.Thread.run(Thread.java:1583) ~[?:?]
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at sun.security.ssl.Alert.createSSLException(Alert.java:130) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:378) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:321) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:316) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1318) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1195) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1138) ~[?:?]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:393) ~[?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:476) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1273) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1260) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:714) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1205) ~[?:?]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1559) ~[?:?]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1405) ~[?:?]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246) ~[?:?]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[?:?]
        ... 16 more
Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:318) ~[?:?]
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:267) ~[?:?]
        at sun.security.validator.Validator.validate(Validator.java:256) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:284) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144) ~[?:?]
        at org.elasticsearch.common.ssl.DiagnosticTrustManager.checkServerTrusted(DiagnosticTrustManager.java:100) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1296) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1195) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1138) ~[?:?]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:393) ~[?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:476) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1273) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1260) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:714) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1205) ~[?:?]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1559) ~[?:?]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1405) ~[?:?]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246) ~[?:?]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[?:?]
        ... 16 more
Caused by: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:157) ~[?:?]
        at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:83) ~[?:?]
        at java.security.cert.CertPathValidator.validate(CertPathValidator.java:309) ~[?:?]
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:313) ~[?:?]
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:267) ~[?:?]
        at sun.security.validator.Validator.validate(Validator.java:256) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:284) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144) ~[?:?]
        at org.elasticsearch.common.ssl.DiagnosticTrustManager.checkServerTrusted(DiagnosticTrustManager.java:100) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1296) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1195) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1138) ~[?:?]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:393) ~[?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:476) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1273) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1260) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:714) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1205) ~[?:?]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1559) ~[?:?]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1405) ~[?:?]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246) ~[?:?]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[?:?]
        ... 16 more
[2024-03-19T10:03:35,161][WARN ][o.e.t.SniffConnectionStrategy] [master-node1] fetching nodes from external cluster [remote-cluster] failed
org.elasticsearch.transport.ConnectTransportException: [][132.17.64.68:9300] connect_exception
        at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1156) ~[elasticsearch-8.11.4.jar:?]
        at org.elasticsearch.action.support.SubscribableListener$FailureResult.complete(SubscribableListener.java:322) ~[elasticsearch-8.11.4.jar:?]
        at org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:234) ~[elasticsearch-8.11.4.jar:?]
        at org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:259) ~[elasticsearch-8.11.4.jar:?]
        at org.elasticsearch.action.support.SubscribableListener.onFailure(SubscribableListener.java:178) ~[elasticsearch-8.11.4.jar:?]
        at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$addListener$0(Netty4TcpChannel.java:62) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[?:?]
        at org.elasticsearch.xpack.core.security.transport.netty4.SecurityNetty4Transport$ClientSslHandlerInitializer.lambda$connect$0(SecurityNetty4Transport.java:352) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[?:?]
        at io.netty.handler.ssl.SslHandler.handleUnwrapThrowable(SslHandler.java:1269) ~[?:?]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1251) ~[?:?]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
        at java.lang.Thread.run(Thread.java:1583) ~[?:?]
Caused by: org.elasticsearch.common.util.concurrent.UncategorizedExecutionException: Failed execution
        at org.elasticsearch.action.support.SubscribableListener.wrapAsExecutionException(SubscribableListener.java:215) ~[elasticsearch-8.11.4.jar:?]
        at org.elasticsearch.common.util.concurrent.ListenableFuture.wrapException(ListenableFuture.java:38) ~[elasticsearch-8.11.4.jar:?]
        at org.elasticsearch.common.util.concurrent.ListenableFuture.wrapException(ListenableFuture.java:27) ~[elasticsearch-8.11.4.jar:?]
        ... 38 more
Caused by: java.util.concurrent.ExecutionException: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at org.elasticsearch.action.support.SubscribableListener.wrapAsExecutionException(SubscribableListener.java:215) ~[elasticsearch-8.11.4.jar:?]
        at org.elasticsearch.common.util.concurrent.ListenableFuture.wrapException(ListenableFuture.java:38) ~[elasticsearch-8.11.4.jar:?]
        at org.elasticsearch.common.util.concurrent.ListenableFuture.wrapException(ListenableFuture.java:27) ~[elasticsearch-8.11.4.jar:?]
        ... 38 more
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at sun.security.ssl.Alert.createSSLException(Alert.java:130) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:378) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:321) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:316) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1318) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1195) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1138) ~[?:?]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:393) ~[?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:476) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1273) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1260) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:714) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1205) ~[?:?]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1559) ~[?:?]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1405) ~[?:?]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246) ~[?:?]
        ... 19 more


Master node config of remote cluster [single node cluster]:
====================================
#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically
# generated to configure Elasticsearch security features on 28-02-2024 11:35:38
#
# --------------------------------------------------------------------------------

# Enable security features
xpack.security.enabled: true

xpack.security.enrollment.enabled: true
xpack.security.authc.api_key.enabled: true
# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: true
  verification_mode: certificate
  certificate: certs/elastic/elastic.crt
  key: certs/elastic/elastic.key
  certificate_authorities: certs/ca/ca.crt


# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
    #truststore.path: certs/cluster_68_certificate_and_key.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
#cluster.initial_master_nodes: ["remote-masternode1"]

# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0

# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
#transport.host: 0.0.0.0

Master node config of local cluster:
==================================
#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically
# generated to configure Elasticsearch security features on 16-02-2024 13:00:13
#
# --------------------------------------------------------------------------------

# Enable security features
xpack.security.enabled: true

xpack.security.autoconfiguration.enabled: true
xpack.security.authc.api_key.enabled: true
xpack.security.enrollment.enabled: true

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: true
  certificate: certs/elastic/elastic.crt
  key: certs/elastic/elastic.key
  certificate_authorities: certs/ca/ca.crt
 
# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12

# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
cluster.initial_master_nodes: ["master-node1"]

# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0

# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
#transport.host: 0.0.0.0

TimV · March 20, 2024, 5:18am

The "local cluster" doesn't trust the certificate provided by the "remote cluster".

I have used the same CA of local cluster to generate the certificates of the remote cluster node

it doesn't look like that is actually true (if it were, you wouldn't see these particular errors).
How exactly did you do that?

Subrahmanyam_Veerank · March 20, 2024, 5:34am

i just copied the ca file of the local cluster and copied it in the remote cluster /etc/elasticsearch/certs/ folder. But i havent changed the transport certificates which are by default ones.

Sir..can you please just share the complete process for establishing the trust between the clusters. i have read the documentation but didnt understand much.

TimV · March 20, 2024, 6:49am

All I would be doing is sharing the same documentation. If you've read it and didn't understand it then sharing it again won't help.

You originally said

I have used the same CA of local cluster to generate the certificates of the remote cluster node

Copying a file and using it to generate certificates are very different things.

What would you like to do now?
Would you prefer to go back to the beginning and regenerate the certificates using the same CA, or update your cluster configuration to trust both CAs?

Subrahmanyam_Veerank · March 20, 2024, 11:58am

i would prefer to update the cluster config to trust both CAs

TimV · March 22, 2024, 12:17am

You'll need to use keytool, which is a command that ships with Java.

Elasticsearch ships with a bundle JavaVM, including the keytool command. Look for a jdk/bin directory in your Elasticsearch installation directory.

Step 1:

One 1 node from "local cluster", determine the password for the transport.p12 keystore:

elasticsearch-keystore show xpack.security.transport.ssl.keystore.secure_password

Step 2:

One that same node, export your transport CA

keytool -exportcert -rfc -alias transport_ca -keystore config/certs/transport.p12 -storepass "THE_PASSWORD_FROM_STEP1" > config/certs/transport_ca-cluster1.crt

Step 3:

One 1 node from "remote cluster", determine the password for the transport.p12 keystore:

elasticsearch-keystore show xpack.security.transport.ssl.keystore.secure_password

Step 4:

One that same node, export your transport CA

keytool -exportcert -rfc -alias transport_ca -keystore config/certs/transport.p12 -storepass "THE_PASSWORD_FROM_STEP3" > config/certs/transport_ca-cluster2.crt

Step 5

On every node in the "local cluster" (cluster 1), import the CA from "remote cluster" (cluster 2) into the transport truststore

keytool -importcert -file /path/to/transport_ca-cluster2.crt -keystore config/certs/transport.p12 -storepass "PASSWORD_FROM_STEP1" -alias "ccs_remote_ca"

Step 6

On every node in the "remote cluster" (cluster 2), import the CA from "local cluster" (cluster 1) into the transport truststore

keytool -importcert -file /path/to/transport_ca-cluster1.crt -keystore config/certs/transport.p12 -storepass "PASSWORD_FROM_STEP3" -alias "ccs_local_ca"

Subrahmanyam_Veerank · March 27, 2024, 12:06pm

Thanks sir

system · April 24, 2024, 12:07pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Cannot establish secure (SSL/TLS) connection between elasticsearch clusters Elasticsearch ccs-cross-cluster-search	3	1363	April 23, 2019
Remote cluster connected but not listed Elasticsearch	2	864	August 7, 2020
Connect to remote cluster Elasticsearch ccs-cross-cluster-search	3	526	January 9, 2023
Remote clusters authentication Elasticsearch ccr-cross-cluster-replication	23	2148	December 9, 2020
Set up remote cluster CA certificate in Elasticsearch 8.8.1 Elasticsearch	1	109	February 23, 2024