# Unable to establish the connection between two clusters

**URL:** <https://discuss.elastic.co/t/unable-to-establish-the-connection-between-two-clusters/354499>\
**Category:** Elasticsearch\
**Tags:** ccs-cross-cluster-search\
**Created:** [March 1, 2024, 5:07am UTC](https://discuss.elastic.co/t/unable-to-establish-the-connection-between-two-clusters/354499 "2024-03-01T05:07:58Z")\
**Posts on this page:** 16\
**Page:** 1

<div class="post-metadata">

**Author:** ![Subrahmanyam\_Veerank](https://avatars.discourse-cdn.com/v4/letter/s/d78d45/32.png) [@Subrahmanyam\_Veerank](https://discuss.elastic.co/u/Subrahmanyam_Veerank)\
**Post date:** [March 1, 2024, 5:07am UTC](https://discuss.elastic.co/t/unable-to-establish-the-connection-between-two-clusters/354499/1 "2024-03-01T05:07:58Z")

</div>

Hi sir,

im trying to establish the connection between the two clusters (local and remote) and i have used the same CA of local cluster to generate the certificates of the remote cluster node. In kibana remote clusters, i have added the details of ip address:port number under seeds and remaining things as by default only. But im unable to establish the connection with remote cluster and when i checked in the logs of remote cluster node it was getting the certificate trust issue. Is there anything that i was missing?

# setup details:

local cluster: 1 master and 3 (data & ingest)  
remote cluster: 1 node (having all the roles)

# Request:

PUT \_cluster/settings  
{  
"persistent": {  
"cluster": {  
"remote": {  
"remote-cluster": {  
"skip\_unavailable": false,  
"mode": "sniff",  
"proxy\_address": null,  
"proxy\_socket\_connections": null,  
"server\_name": null,  
"seeds": [  
"xx.xxx.xxx.xxx:9300"  
],  
"node\_connections": 3  
}  
}  
}  
}  
}

---

<div class="post-metadata">

**Author:** ![rtwolfe94022](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rtwolfe94022/32/132253_2.png) [@rtwolfe94022](https://discuss.elastic.co/u/rtwolfe94022)\
**Post date:** [March 1, 2024, 5:12am UTC](https://discuss.elastic.co/t/unable-to-establish-the-connection-between-two-clusters/354499/2 "2024-03-01T05:12:36Z")

</div>

To resolve the certificate trust issue between your local and remote Elasticsearch clusters, ensure the following:

1. **Both clusters trust the CA** : The CA used to generate the node certificates must be imported into the truststores of both clusters.
2. **Certificates have correct SAN entries** : Check that the certificates include the correct Subject Alternative Names (SANs) for IP addresses or DNS names.
3. **SSL is enabled and correctly configured** : Verify `xpack.security.transport.ssl` settings in `elasticsearch.yml` for both clusters, ensuring paths to certificates and keys are correct.
4. **Firewall and network connectivity** : Ensure no network issues or firewalls block the connection on the transport port (default 9300).
5. **Review logs for specific errors** : Check Elasticsearch logs for detailed SSL or connection error messages.

Ensure the setup is correctly pointing to the certificate files and that all configurations align with security requirements.

---

<div class="post-metadata">

**Author:** ![Subrahmanyam\_Veerank](https://avatars.discourse-cdn.com/v4/letter/s/d78d45/32.png) [@Subrahmanyam\_Veerank](https://discuss.elastic.co/u/Subrahmanyam_Veerank)\
**Post date:** [March 1, 2024, 5:15am UTC](https://discuss.elastic.co/t/unable-to-establish-the-connection-between-two-clusters/354499/3 "2024-03-01T05:15:06Z")

</div>

> [@rtwolfe94022](#):
>
> **Both clusters trust the CA** : The CA used to generate the node certificates must be imported into the truststores of both clusters.

sir..how to import into the truststore?

---

<div class="post-metadata">

**Author:** ![Subrahmanyam\_Veerank](https://avatars.discourse-cdn.com/v4/letter/s/d78d45/32.png) [@Subrahmanyam\_Veerank](https://discuss.elastic.co/u/Subrahmanyam_Veerank)\
**Post date:** [March 1, 2024, 8:33am UTC](https://discuss.elastic.co/t/unable-to-establish-the-connection-between-two-clusters/354499/4 "2024-03-01T08:33:02Z")

</div>

sir... i have generated ca.crt and ca.key on the local cluster previously using below command.

./elasticsearch-certutil ca --pem --out /etc/elasticsearch/certs/ca.zip

now i have copied the ca.crt and ca.key to the remote cluster /etc/elasticsearch/certs . Kindly share the commands to make import into the truststores to establish the trust.

# Enable encryption and mutual authentication between cluster nodes

xpack.security.transport.ssl:  
enabled: true  
verification\_mode: certificate  
keystore.path: certs/transport.p12  
truststore.path: certs/transport.p12

the transport.ssl is by default on both the clusters..

---

<div class="post-metadata">

**Author:** ![Subrahmanyam\_Veerank](https://avatars.discourse-cdn.com/v4/letter/s/d78d45/32.png) [@Subrahmanyam\_Veerank](https://discuss.elastic.co/u/Subrahmanyam_Veerank)\
**Post date:** [March 7, 2024, 4:27am UTC](https://discuss.elastic.co/t/unable-to-establish-the-connection-between-two-clusters/354499/5 "2024-03-07T04:27:19Z")

</div>

> [@rtwolfe94022](#):
>
> **Both clusters trust the CA** : The CA used to generate the node certificates must be imported into the truststores of both clusters.

sir..can you please share the procedure?

---

<div class="post-metadata">

**Author:** ![Subrahmanyam\_Veerank](https://avatars.discourse-cdn.com/v4/letter/s/d78d45/32.png) [@Subrahmanyam\_Veerank](https://discuss.elastic.co/u/Subrahmanyam_Veerank)\
**Post date:** [March 14, 2024, 8:57am UTC](https://discuss.elastic.co/t/unable-to-establish-the-connection-between-two-clusters/354499/6 "2024-03-14T08:57:45Z")

</div>

kindly help us in this regard?

---

<div class="post-metadata">

**Author:** ![Subrahmanyam\_Veerank](https://avatars.discourse-cdn.com/v4/letter/s/d78d45/32.png) [@Subrahmanyam\_Veerank](https://discuss.elastic.co/u/Subrahmanyam_Veerank)\
**Post date:** [March 18, 2024, 9:38am UTC](https://discuss.elastic.co/t/unable-to-establish-the-connection-between-two-clusters/354499/7 "2024-03-18T09:38:30Z")

</div>

Can anyone help me regarding this issue?

---

<div class="post-metadata">

**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\
**Post date:** [March 19, 2024, 2:35am UTC](https://discuss.elastic.co/t/unable-to-establish-the-connection-between-two-clusters/354499/8 "2024-03-19T02:35:14Z")

</div>

> [@Subrahmanyam\_Veerank](#):
>
> when i checked in the logs of remote cluster node it was getting the certificate trust issue.

Please show the full log, including the SSL diagnostic log message, and also your most recent configuration.

---

<div class="post-metadata">

**Author:** ![Subrahmanyam\_Veerank](https://avatars.discourse-cdn.com/v4/letter/s/d78d45/32.png) [@Subrahmanyam\_Veerank](https://discuss.elastic.co/u/Subrahmanyam_Veerank)\
**Post date:** [March 19, 2024, 4:49am UTC](https://discuss.elastic.co/t/unable-to-establish-the-connection-between-two-clusters/354499/9 "2024-03-19T04:49:15Z")

</div>

Sir please find the logs and SSL related configurations below. Im using the same CA [Self signed] for both the clusters.

```auto
Remote cluster settings in kibana [local cluster]:
=================================
PUT _cluster/settings
{
  "persistent": {
    "cluster": {
      "remote": {
        "remote-cluster": {
          "skip_unavailable": false,
          "mode": "sniff",
          "proxy_address": null,
          "proxy_socket_connections": null,
          "server_name": null,
          "seeds": [
            "132.17.64.68:9300"
          ],
          "node_connections": 3
        }
      }
    }
  }
}

Remote cluster logs:
===================

[2024-03-19T10:02:12,802][INFO][o.e.m.j.JvmGcMonitorService] [node-1] [gc][177] overhead, spent [301ms] collecting in the last [1s]
[2024-03-19T10:03:35,800][WARN][o.e.x.c.s.t.n.SecurityNetty4Transport] [node-1] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/132.17.64.68:9300, remoteAddress=/122.17.14.63:50516, profile=default}
[2024-03-19T10:03:35,825][WARN][o.e.x.c.s.t.n.SecurityNetty4Transport] [node-1] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/132.17.64.68:9300, remoteAddress=/122.17.14.63:50520, profile=default}
[2024-03-19T10:03:35,850][WARN][o.e.x.c.s.t.n.SecurityNetty4Transport] [node-1] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/132.17.64.68:9300, remoteAddress=/122.17.14.64:45184, profile=default}
[2024-03-19T10:03:35,927][WARN][o.e.x.c.s.t.n.SecurityNetty4Transport] [node-1] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/132.17.64.68:9300, remoteAddress=/122.17.14.63:50518, profile=default}
[2024-03-19T10:03:36,275][WARN][o.e.x.c.s.t.n.SecurityNetty4Transport] [node-1] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=/132.17.64.68:9300, remoteAddress=/122.17.14.70:37968, profile=default}

Local cluster logs:
==================

[2024-03-19T10:03:34,831][INFO][o.e.c.s.ClusterSettings] [master-node1] updating [cluster.remote.remote-cluster.mode] from [SNIFF] to [sniff]
[2024-03-19T10:03:34,832][INFO][o.e.c.s.ClusterSettings] [master-node1] updating [cluster.remote.remote-cluster.seeds] from [[]] to [["132.17.64.68:9300"]]
[2024-03-19T10:03:34,832][INFO][o.e.c.s.ClusterSettings] [master-node1] updating [cluster.remote.remote-cluster.mode] from [SNIFF] to [sniff]
[2024-03-19T10:03:34,833][INFO][o.e.c.s.ClusterSettings] [master-node1] updating [cluster.remote.remote-cluster.seeds] from [[]] to [["132.17.64.68:9300"]]
[2024-03-19T10:03:35,152][WARN][o.e.c.s.DiagnosticTrustManager] [master-node1] failed to establish trust with server at [<unknown host>]; the server provided a certificate with subject name [CN=remote-masternode1], fingerprint [039575ec662d29bc50bcbeb939bab407f52ef788], no keyUsage and no extendedKeyUsage; the certificate is valid between [2024-02-28T11:35:46Z] and [2123-02-04T11:35:46Z] (current time is [2024-03-19T04:33:35.152130606Z], certificate dates are valid); the session uses cipher suite [TLS_AES_256_GCM_SHA384] and protocol [TLSv1.3]; the certificate does not have any subject alternative names; the certificate is issued by [CN=Elasticsearch security auto-configuration HTTP CA]; the certificate is signed by (subject [CN=Elasticsearch security auto-configuration HTTP CA] fingerprint [88cd0259c526e732dfc3c4a6c7eba8ffd]) which is self-issued; the [CN=Elasticsearch security auto-configuration HTTP CA] certificate is not trusted in this ssl context ([xpack.security.transport.ssl (with trust configuration: StoreTrustConfig{path=certs/transport.p12, password=<non-empty>, type=PKCS12, algorithm=PKIX})]); this ssl context does trust a certificate with subject [CN=Elasticsearch security auto-configuration HTTP CA] but the trusted certificate has fingerprint [577a707106cd8df5c4cdc48c06b19e25]
sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:318) ~[?:?]
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:267) ~[?:?]
        at sun.security.validator.Validator.validate(Validator.java:256) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:284) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144) ~[?:?]
        at org.elasticsearch.common.ssl.DiagnosticTrustManager.checkServerTrusted(DiagnosticTrustManager.java:100) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1296) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1195) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1138) ~[?:?]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:393) ~[?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:476) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1273) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1260) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:714) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1205) ~[?:?]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1559) ~[?:?]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1405) ~[?:?]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246) ~[?:?]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
        at java.lang.Thread.run(Thread.java:1583) ~[?:?]
Caused by: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:157) ~[?:?]
        at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:83) ~[?:?]
        at java.security.cert.CertPathValidator.validate(CertPathValidator.java:309) ~[?:?]
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:313) ~[?:?]
        ... 36 more
[2024-03-19T10:03:35,162][WARN][o.e.t.TcpTransport] [master-node1] exception caught on transport layer [Netty4TcpChannel{localAddress=/122.17.14.70:37968, remoteAddress=/132.17.64.68:9300, profile=default}], closing connection
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:499) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
        at java.lang.Thread.run(Thread.java:1583) ~[?:?]
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at sun.security.ssl.Alert.createSSLException(Alert.java:130) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:378) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:321) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:316) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1318) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1195) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1138) ~[?:?]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:393) ~[?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:476) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1273) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1260) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:714) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1205) ~[?:?]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1559) ~[?:?]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1405) ~[?:?]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246) ~[?:?]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[?:?]
        ... 16 more
Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:318) ~[?:?]
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:267) ~[?:?]
        at sun.security.validator.Validator.validate(Validator.java:256) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:284) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144) ~[?:?]
        at org.elasticsearch.common.ssl.DiagnosticTrustManager.checkServerTrusted(DiagnosticTrustManager.java:100) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1296) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1195) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1138) ~[?:?]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:393) ~[?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:476) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1273) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1260) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:714) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1205) ~[?:?]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1559) ~[?:?]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1405) ~[?:?]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246) ~[?:?]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[?:?]
        ... 16 more
Caused by: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:157) ~[?:?]
        at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:83) ~[?:?]
        at java.security.cert.CertPathValidator.validate(CertPathValidator.java:309) ~[?:?]
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:313) ~[?:?]
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:267) ~[?:?]
        at sun.security.validator.Validator.validate(Validator.java:256) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:284) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144) ~[?:?]
        at org.elasticsearch.common.ssl.DiagnosticTrustManager.checkServerTrusted(DiagnosticTrustManager.java:100) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1296) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1195) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1138) ~[?:?]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:393) ~[?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:476) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1273) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1260) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:714) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1205) ~[?:?]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1559) ~[?:?]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1405) ~[?:?]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246) ~[?:?]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[?:?]
        ... 16 more
[2024-03-19T10:03:35,161][WARN][o.e.t.SniffConnectionStrategy] [master-node1] fetching nodes from external cluster [remote-cluster] failed
org.elasticsearch.transport.ConnectTransportException: [][132.17.64.68:9300] connect_exception
        at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1156) ~[elasticsearch-8.11.4.jar:?]
        at org.elasticsearch.action.support.SubscribableListener$FailureResult.complete(SubscribableListener.java:322) ~[elasticsearch-8.11.4.jar:?]
        at org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:234) ~[elasticsearch-8.11.4.jar:?]
        at org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:259) ~[elasticsearch-8.11.4.jar:?]
        at org.elasticsearch.action.support.SubscribableListener.onFailure(SubscribableListener.java:178) ~[elasticsearch-8.11.4.jar:?]
        at org.elasticsearch.transport.netty4.Netty4TcpChannel.lambda$addListener$0(Netty4TcpChannel.java:62) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[?:?]
        at org.elasticsearch.xpack.core.security.transport.netty4.SecurityNetty4Transport$ClientSslHandlerInitializer.lambda$connect$0(SecurityNetty4Transport.java:352) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629) ~[?:?]
        at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118) ~[?:?]
        at io.netty.handler.ssl.SslHandler.handleUnwrapThrowable(SslHandler.java:1269) ~[?:?]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1251) ~[?:?]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
        at java.lang.Thread.run(Thread.java:1583) ~[?:?]
Caused by: org.elasticsearch.common.util.concurrent.UncategorizedExecutionException: Failed execution
        at org.elasticsearch.action.support.SubscribableListener.wrapAsExecutionException(SubscribableListener.java:215) ~[elasticsearch-8.11.4.jar:?]
        at org.elasticsearch.common.util.concurrent.ListenableFuture.wrapException(ListenableFuture.java:38) ~[elasticsearch-8.11.4.jar:?]
        at org.elasticsearch.common.util.concurrent.ListenableFuture.wrapException(ListenableFuture.java:27) ~[elasticsearch-8.11.4.jar:?]
        ... 38 more
Caused by: java.util.concurrent.ExecutionException: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at org.elasticsearch.action.support.SubscribableListener.wrapAsExecutionException(SubscribableListener.java:215) ~[elasticsearch-8.11.4.jar:?]
        at org.elasticsearch.common.util.concurrent.ListenableFuture.wrapException(ListenableFuture.java:38) ~[elasticsearch-8.11.4.jar:?]
        at org.elasticsearch.common.util.concurrent.ListenableFuture.wrapException(ListenableFuture.java:27) ~[elasticsearch-8.11.4.jar:?]
        ... 38 more
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at sun.security.ssl.Alert.createSSLException(Alert.java:130) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:378) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:321) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:316) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1318) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1195) ~[?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1138) ~[?:?]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:393) ~[?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:476) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1273) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1260) ~[?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:714) ~[?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1205) ~[?:?]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1559) ~[?:?]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1405) ~[?:?]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246) ~[?:?]
        ... 19 more

Master node config of remote cluster [single node cluster]:
====================================
#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically
# generated to configure Elasticsearch security features on 28-02-2024 11:35:38
#
# --------------------------------------------------------------------------------

# Enable security features
xpack.security.enabled: true

xpack.security.enrollment.enabled: true
xpack.security.authc.api_key.enabled: true
# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: true
  verification_mode: certificate
  certificate: certs/elastic/elastic.crt
  key: certs/elastic/elastic.key
  certificate_authorities: certs/ca/ca.crt

# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12
    #truststore.path: certs/cluster_68_certificate_and_key.p12
# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
#cluster.initial_master_nodes: ["remote-masternode1"]

# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0

# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
#transport.host: 0.0.0.0

Master node config of local cluster:
==================================
#----------------------- BEGIN SECURITY AUTO CONFIGURATION -----------------------
#
# The following settings, TLS certificates, and keys have been automatically
# generated to configure Elasticsearch security features on 16-02-2024 13:00:13
#
# --------------------------------------------------------------------------------

# Enable security features
xpack.security.enabled: true

xpack.security.autoconfiguration.enabled: true
xpack.security.authc.api_key.enabled: true
xpack.security.enrollment.enabled: true

# Enable encryption for HTTP API client connections, such as Kibana, Logstash, and Agents
xpack.security.http.ssl:
  enabled: true
  certificate: certs/elastic/elastic.crt
  key: certs/elastic/elastic.key
  certificate_authorities: certs/ca/ca.crt
 
# Enable encryption and mutual authentication between cluster nodes
xpack.security.transport.ssl:
  enabled: true
  verification_mode: certificate
  keystore.path: certs/transport.p12
  truststore.path: certs/transport.p12

# Create a new cluster with the current node only
# Additional nodes can still join the cluster later
cluster.initial_master_nodes: ["master-node1"]

# Allow HTTP API connections from anywhere
# Connections are encrypted and require user authentication
http.host: 0.0.0.0

# Allow other nodes to join the cluster from anywhere
# Connections are encrypted and mutually authenticated
#transport.host: 0.0.0.0

```

---

<div class="post-metadata">

**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\
**Post date:** [March 20, 2024, 5:18am UTC](https://discuss.elastic.co/t/unable-to-establish-the-connection-between-two-clusters/354499/10 "2024-03-20T05:18:37Z")

</div>

The "local cluster" doesn't trust the certificate provided by the "remote cluster".

> I have used the same CA of local cluster to generate the certificates of the remote cluster node

it doesn't look like that is actually true (if it were, you wouldn't see these particular errors).  
How exactly did you do that?

---

<div class="post-metadata">

**Author:** ![Subrahmanyam\_Veerank](https://avatars.discourse-cdn.com/v4/letter/s/d78d45/32.png) [@Subrahmanyam\_Veerank](https://discuss.elastic.co/u/Subrahmanyam_Veerank)\
**Post date:** [March 20, 2024, 5:34am UTC](https://discuss.elastic.co/t/unable-to-establish-the-connection-between-two-clusters/354499/11 "2024-03-20T05:34:17Z")

</div>

i just copied the ca file of the local cluster and copied it in the remote cluster /etc/elasticsearch/certs/ folder. But i havent changed the transport certificates which are by default ones.

Sir..can you please just share the complete process for establishing the trust between the clusters. i have read the documentation but didnt understand much.

---

<div class="post-metadata">

**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\
**Post date:** [March 20, 2024, 6:49am UTC](https://discuss.elastic.co/t/unable-to-establish-the-connection-between-two-clusters/354499/12 "2024-03-20T06:49:32Z")

</div>

> [@Subrahmanyam\_Veerank](#):
>
> Sir..can you please just share the complete process for establishing the trust between the clusters. i have read the documentation but didnt understand much.

All I would be doing is sharing the same documentation. If you've read it and didn't understand it then sharing it again won't help.

> [@Subrahmanyam\_Veerank](#):
>
> i just copied the ca file of the local cluster and copied it in the remote cluster /etc/elasticsearch/certs/ folder

You originally said

> I have used the same CA of local cluster to generate the certificates of the remote cluster node

Copying a file and using it to generate certificates are very different things.

What would you like to do now?  
Would you prefer to go back to the beginning and regenerate the certificates using the same CA, or update your cluster configuration to trust both CAs?

---

<div class="post-metadata">

**Author:** ![Subrahmanyam\_Veerank](https://avatars.discourse-cdn.com/v4/letter/s/d78d45/32.png) [@Subrahmanyam\_Veerank](https://discuss.elastic.co/u/Subrahmanyam_Veerank)\
**Post date:** [March 20, 2024, 11:58am UTC](https://discuss.elastic.co/t/unable-to-establish-the-connection-between-two-clusters/354499/13 "2024-03-20T11:58:03Z")

</div>

> [@TimV](#):
>
> Would you prefer to go back to the beginning and regenerate the certificates using the same CA, or update your cluster configuration to trust both CAs?

i would prefer to update the cluster config to trust both CAs

---

<div class="post-metadata">

**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\
**Post date:** [March 22, 2024, 12:17am UTC](https://discuss.elastic.co/t/unable-to-establish-the-connection-between-two-clusters/354499/14 "2024-03-22T00:17:38Z")

</div>

You'll need to use `keytool`, which is a command that ships with Java.

Elasticsearch ships with a bundle JavaVM, including the `keytool` command. Look for a `jdk/bin` directory in your Elasticsearch installation directory.

### Step 1:

One 1 node from "local cluster", determine the password for the `transport.p12` keystore:

```auto
elasticsearch-keystore show xpack.security.transport.ssl.keystore.secure_password

```

### Step 2:

One that same node, export your transport CA

```auto
keytool -exportcert -rfc -alias transport_ca -keystore config/certs/transport.p12 -storepass "THE_PASSWORD_FROM_STEP1" > config/certs/transport_ca-cluster1.crt

```

### Step 3:

One 1 node from "remote cluster", determine the password for the `transport.p12` keystore:

```auto
elasticsearch-keystore show xpack.security.transport.ssl.keystore.secure_password

```

### Step 4:

One that same node, export your transport CA

```auto
keytool -exportcert -rfc -alias transport_ca -keystore config/certs/transport.p12 -storepass "THE_PASSWORD_FROM_STEP3" > config/certs/transport_ca-cluster2.crt

```

### Step 5

On _every_ node in the "local cluster" (cluster 1), import the CA from "remote cluster" (cluster 2) into the transport truststore

```auto
keytool -importcert -file /path/to/transport_ca-cluster2.crt -keystore config/certs/transport.p12 -storepass "PASSWORD_FROM_STEP1" -alias "ccs_remote_ca"

```

### Step 6

On _every_ node in the "remote cluster" (cluster 2), import the CA from "local cluster" (cluster 1) into the transport truststore

```auto
keytool -importcert -file /path/to/transport_ca-cluster1.crt -keystore config/certs/transport.p12 -storepass "PASSWORD_FROM_STEP3" -alias "ccs_local_ca"

```

---

<div class="post-metadata">

**Author:** ![Subrahmanyam\_Veerank](https://avatars.discourse-cdn.com/v4/letter/s/d78d45/32.png) [@Subrahmanyam\_Veerank](https://discuss.elastic.co/u/Subrahmanyam_Veerank)\
**Post date:** [March 27, 2024, 12:06pm UTC](https://discuss.elastic.co/t/unable-to-establish-the-connection-between-two-clusters/354499/15 "2024-03-27T12:06:51Z")

</div>

Thanks sir

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 24, 2024, 12:07pm UTC](https://discuss.elastic.co/t/unable-to-establish-the-connection-between-two-clusters/354499/16 "2024-04-24T12:07:00Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.