# Remote clusters authentication

**URL:** <https://discuss.elastic.co/t/remote-clusters-authentication/254988>\
**Category:** Elasticsearch\
**Tags:** ccr-cross-cluster-replication\
**Created:** [November 10, 2020, 10:59pm UTC](https://discuss.elastic.co/t/remote-clusters-authentication/254988 "2020-11-10T22:59:17Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![kyle\_che](https://avatars.discourse-cdn.com/v4/letter/k/ce73a5/32.png) [@kyle\_che](https://discuss.elastic.co/u/kyle_che)\
**Post date:** [November 10, 2020, 10:59pm UTC](https://discuss.elastic.co/t/remote-clusters-authentication/254988/1 "2020-11-10T22:59:17Z")

</div>

I am trying to test out remote clusters and then cross cluster replication. When inputting information about the other cluster under "Remote Clusters", Connection is saying "Not connected". I'm guessing this is due to my clusters having passwords set. Is there a way to pass the username/password so that the connection status will say connected?

---

<div class="post-metadata">

**Author:** ![kyle\_che](https://avatars.discourse-cdn.com/v4/letter/k/ce73a5/32.png) [@kyle\_che](https://discuss.elastic.co/u/kyle_che)\
**Post date:** [November 10, 2020, 11:00pm UTC](https://discuss.elastic.co/t/remote-clusters-authentication/254988/2 "2020-11-10T23:00:27Z")

</div>

This is what it currently looks like.

![remote_clusters](https://us1.discourse-cdn.com/elastic/original/3X/b/8/b8b66b12f99266ca22a99f1eccffb3b4639a8fb9.png)

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [November 10, 2020, 11:03pm UTC](https://discuss.elastic.co/t/remote-clusters-authentication/254988/3 "2020-11-10T23:03:08Z")

</div>

The cluster trust isn't setup as a user/password approach. [https://www.elastic.co/guide/en/elasticsearch/reference/current/cross-cluster-configuring.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/cross-cluster-configuring.html) explains it more.

What do your Elasticsearch logs show?

---

<div class="post-metadata">

**Author:** ![kyle\_che](https://avatars.discourse-cdn.com/v4/letter/k/ce73a5/32.png) [@kyle\_che](https://discuss.elastic.co/u/kyle_che)\
**Post date:** [November 10, 2020, 11:24pm UTC](https://discuss.elastic.co/t/remote-clusters-authentication/254988/4 "2020-11-10T23:24:26Z")

</div>

```auto
[2020-11-10T22:11:04,656][WARN][o.e.c.s.DiagnosticTrustManager] [d-gp2-kyleesdb1-1] failed to establish trust with server at [<unknown host>]; the server provided a certificate with subject name [CN=instance] and fingerprint [d215f08f8d81045fc228d607b5abe4c6e65b4b19]; the certificate does not have any subject alternative names; the certificate is issued by [CN=Elastic Certificate Tool Autogenerated CA]; the certificate is signed by (subject [CN=Elastic Certificate Tool Autogenerated CA] fingerprint [91ba3f59359417423c3442eda6bea3cb5e9af46c]) which is self-issued; the [CN=Elastic Certificate Tool Autogenerated CA] certificate is not trusted in this ssl context ([xpack.security.transport.ssl]); this ssl context does trust a certificate with subject [CN=Elastic Certificate Tool Autogenerated CA] but the trusted certificate has fingerprint [a1116e365cf3e89c41a907ce46c96585e9d9afb0]
sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:369) ~[?:?]
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:275) ~[?:?]
        at sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144) ~[?:?]
        at org.elasticsearch.common.ssl.DiagnosticTrustManager.checkServerTrusted(DiagnosticTrustManager.java:110) [elasticsearch-ssl-config-7.9.3.jar:7.9.3]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1318) [?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1215) [?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1158) [?:?]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) [?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) [?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1267) [?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1254) [?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:691) [?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1199) [?:?]
        at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1542) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1556) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1440) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
        at java.lang.Thread.run(Thread.java:832) [?:?]
Caused by: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:158) ~[?:?]
        at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:84) ~[?:?]
        at java.security.cert.CertPathValidator.validate(CertPathValidator.java:309) ~[?:?]
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:364) ~[?:?]
        ... 37 more

```

---

<div class="post-metadata">

**Author:** ![kyle\_che](https://avatars.discourse-cdn.com/v4/letter/k/ce73a5/32.png) [@kyle\_che](https://discuss.elastic.co/u/kyle_che)\
**Post date:** [November 10, 2020, 11:25pm UTC](https://discuss.elastic.co/t/remote-clusters-authentication/254988/5 "2020-11-10T23:25:32Z")

</div>

must both clusters have the same elastic-certificates.p12 file?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [November 10, 2020, 11:26pm UTC](https://discuss.elastic.co/t/remote-clusters-authentication/254988/6 "2020-11-10T23:26:36Z")

</div>

From that doc link;

> Enable a trust relationship between the cluster used for performing cross cluster search (the local cluster) and all remote clusters. This can be done either by:
> 
> - Using the same certificate authority to generate certificates for all connected clusters, or
> - Adding the CA certificate from the local cluster as a trusted CA in each remote cluster (see [Transport TLS settings](https://www.elastic.co/guide/en/elasticsearch/reference/7.9/security-settings.html#transport-tls-ssl-settings)).

---

<div class="post-metadata">

**Author:** ![kyle\_che](https://avatars.discourse-cdn.com/v4/letter/k/ce73a5/32.png) [@kyle\_che](https://discuss.elastic.co/u/kyle_che)\
**Post date:** [November 10, 2020, 11:35pm UTC](https://discuss.elastic.co/t/remote-clusters-authentication/254988/7 "2020-11-10T23:35:16Z")

</div>

that worked. one last question. I have "Remote Clusters" setup. I am missing the " Cross Cluster Replication" option on the left hand side. Does something have to be installed for this feature?

---

<div class="post-metadata">

**Author:** ![kyle\_che](https://avatars.discourse-cdn.com/v4/letter/k/ce73a5/32.png) [@kyle\_che](https://discuss.elastic.co/u/kyle_che)\
**Post date:** [November 10, 2020, 11:39pm UTC](https://discuss.elastic.co/t/remote-clusters-authentication/254988/8 "2020-11-10T23:39:02Z")

</div>

looks like it's not free so it requires a license. thanks for your help.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [November 10, 2020, 11:41pm UTC](https://discuss.elastic.co/t/remote-clusters-authentication/254988/9 "2020-11-10T23:41:38Z")

</div>

You can upgrade a Basic license to a 30 day Platinum trial if you want - [https://www.elastic.co/guide/en/kibana/current/managing-licenses.html](https://www.elastic.co/guide/en/kibana/current/managing-licenses.html).

---

<div class="post-metadata">

**Author:** ![kyle\_che](https://avatars.discourse-cdn.com/v4/letter/k/ce73a5/32.png) [@kyle\_che](https://discuss.elastic.co/u/kyle_che)\
**Post date:** [November 10, 2020, 11:49pm UTC](https://discuss.elastic.co/t/remote-clusters-authentication/254988/10 "2020-11-10T23:49:19Z")

</div>

```auto
ok, did that and i'm getting the following error.

Can't create follower index

[index_not_found_exception] no such index [kibana_sample_data_flights], with { index_uuid="_na_" & index="kibana_sample_data_flights" }

* no such index [kibana_sample_data_flights]

```

 ![Screen Shot 2020-11-10 at 4.48.43 PM](https://us1.discourse-cdn.com/elastic/original/3X/d/9/d9a6487078360e61f8dc7429b6f1271a7fc81499.png)

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [November 10, 2020, 11:50pm UTC](https://discuss.elastic.co/t/remote-clusters-authentication/254988/11 "2020-11-10T23:50:51Z")

</div>

Did you load the sample data? You can do that from the home screen - [https://www.elastic.co/guide/en/kibana/current/tutorial-sample-data.html](https://www.elastic.co/guide/en/kibana/current/tutorial-sample-data.html)

---

<div class="post-metadata">

**Author:** ![kyle\_che](https://avatars.discourse-cdn.com/v4/letter/k/ce73a5/32.png) [@kyle\_che](https://discuss.elastic.co/u/kyle_che)\
**Post date:** [November 10, 2020, 11:51pm UTC](https://discuss.elastic.co/t/remote-clusters-authentication/254988/12 "2020-11-10T23:51:49Z")

</div>

yes, it's been loaded but won't replicate.

```auto
# curl -XGET "elastic:***@localhost:9200/_cat/indices?v&index=kibana*&pretty"
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open kibana_sample_data_ecommerce gGFEHJKAREiFG2MueQ-2OA 1 1 4675 0 9.2mb 4.6mb
green open kibana_sample_data_logs CLc7k_fNQoCJ5cYfpMPZpA 1 1 14074 0 19.1mb 9.6mb
green open kibana_sample_data_flights nBDI26H1T4WNFXr5CGgwkg 1 1 13059 0 11.9mb 6mb

```

---

<div class="post-metadata">

**Author:** ![kyle\_che](https://avatars.discourse-cdn.com/v4/letter/k/ce73a5/32.png) [@kyle\_che](https://discuss.elastic.co/u/kyle_che)\
**Post date:** [November 11, 2020, 12:14am UTC](https://discuss.elastic.co/t/remote-clusters-authentication/254988/13 "2020-11-11T00:14:40Z")

</div>

kibana error is:

```auto
{"type":"response","@timestamp":"2020-11-11T00:14:14Z","tags":[],"pid":15625,"method":"post","statusCode":404,"req":{"url":"/api/cross_cluster_replication/follower_indices","method":"post","headers":{"host":"d-gp2-kyleesdb1-1.imovetv.com:5601","connection":"keep-alive","content-length":"441","kbn-version":"7.9.3","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.183 Safari/537.36","content-type":"application/json","accept":"*/*","origin":"http://d-gp2-kyleesdb1-1.imovetv.com:5601","referer":"http://d-gp2-kyleesdb1-1.imovetv.com:5601/app/management/data/cross_cluster_replication/follower_indices/add","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9,es;q=0.8,mt;q=0.7"},"remoteAddress":"10.124.250.242","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.183 Safari/537.36","referer":"http://d-gp2-kyleesdb1-1.imovetv.com:5601/app/management/data/cross_cluster_replication/follower_indices/add"},"res":{"statusCode":404,"responseTime":142,"contentLength":9},"message":"POST /api/cross_cluster_replication/follower_indices 404 142ms - 9.0B"}

```

---

<div class="post-metadata">

**Author:** ![kyle\_che](https://avatars.discourse-cdn.com/v4/letter/k/ce73a5/32.png) [@kyle\_che](https://discuss.elastic.co/u/kyle_che)\
**Post date:** [November 11, 2020, 12:42am UTC](https://discuss.elastic.co/t/remote-clusters-authentication/254988/14 "2020-11-11T00:42:58Z")

</div>

I have also created a new index testme with 1 doc and it gets the same error. What am i missng?

```auto
Can't create follower index
[index_not_found_exception] no such index [testme], with { index_uuid="_na_" & index="testme" }
no such index [testme]

```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [November 11, 2020, 12:48am UTC](https://discuss.elastic.co/t/remote-clusters-authentication/254988/15 "2020-11-11T00:48:40Z")

</div>

What the output from the `GET /_ccr/stats` endpoint?

---

<div class="post-metadata">

**Author:** ![kyle\_che](https://avatars.discourse-cdn.com/v4/letter/k/ce73a5/32.png) [@kyle\_che](https://discuss.elastic.co/u/kyle_che)\
**Post date:** [November 11, 2020, 12:50am UTC](https://discuss.elastic.co/t/remote-clusters-authentication/254988/16 "2020-11-11T00:50:13Z")

</div>

here is what is the code that is being run...

```auto
curl -XGET "http://d-gp2-kyleesdb1-1.imovetv.com:9200/_cat/indices?v&index=test*&pretty"
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open testme 8j-Sam_gS22BsqENTMMXzA 1 1 1 0 7.6kb 3.8kb

curl -XPUT "http://d-gp2-kyleesdb1-1.imovetv.com:9200/follow_testme/_ccr/follow" -H 'Content-Type: application/json' -d'{ "remote_cluster": "clusterTwo", "leader_index": "testme", "max_read_request_operation_count": 5120, "max_outstanding_read_requests": 12, "max_read_request_size": "32mb", "max_write_request_operation_count": 5120, "max_write_request_size": "9223372036854775807b", "max_outstanding_write_requests": 9, "max_write_buffer_count": 2147483647, "max_write_buffer_size": "512mb", "max_retry_delay": "500ms", "read_poll_timeout": "1m"}'

{
  "error" : {
    "root_cause" : [
      {
        "type" : "index_not_found_exception",
        "reason" : "no such index [testme]",
        "index_uuid" : "_na_",
        "index" : "testme"
      }
    ],
    "type" : "index_not_found_exception",
    "reason" : "no such index [testme]",
    "index_uuid" : "_na_",
    "index" : "testme"
  },
  "status" : 404
}

```

---

<div class="post-metadata">

**Author:** ![kyle\_che](https://avatars.discourse-cdn.com/v4/letter/k/ce73a5/32.png) [@kyle\_che](https://discuss.elastic.co/u/kyle_che)\
**Post date:** [November 11, 2020, 5:55pm UTC](https://discuss.elastic.co/t/remote-clusters-authentication/254988/17 "2020-11-11T17:55:51Z")

</div>

Here is the output from the command you sent me.

```auto
curl -XGET "http://d-gp2-kyleesdb1-1.imovetv.com:9200/_ccr/stats"
{
  "auto_follow_stats" : {
    "number_of_failed_follow_indices" : 0,
    "number_of_failed_remote_cluster_state_requests" : 0,
    "number_of_successful_follow_indices" : 0,
    "recent_auto_follow_errors" : [],
    "auto_followed_clusters" : []
  },
  "follow_stats" : {
    "indices" : []
  }
}

```

---

<div class="post-metadata">

**Author:** ![kyle\_che](https://avatars.discourse-cdn.com/v4/letter/k/ce73a5/32.png) [@kyle\_che](https://discuss.elastic.co/u/kyle_che)\
**Post date:** [November 11, 2020, 6:03pm UTC](https://discuss.elastic.co/t/remote-clusters-authentication/254988/18 "2020-11-11T18:03:35Z")

</div>

just getting the below error.

 ![Screen Shot 2020-11-11 at 11.02.57 AM](https://us1.discourse-cdn.com/elastic/original/3X/e/a/ea3eacc863f6faaefd67a4ea78a4cc25b7fc1814.png)

---

<div class="post-metadata">

**Author:** ![kyle\_che](https://avatars.discourse-cdn.com/v4/letter/k/ce73a5/32.png) [@kyle\_che](https://discuss.elastic.co/u/kyle_che)\
**Post date:** [November 11, 2020, 6:23pm UTC](https://discuss.elastic.co/t/remote-clusters-authentication/254988/19 "2020-11-11T18:23:12Z")

</div>

Tried to do it manually but same error.

```auto
curl -XPUT "elastic:***@d-gp2-kyleesdb1-1.imovetv.com:9200/testme-copy/_ccr/follow?pretty" -H 'Content-Type: application/json' -d'{ "remote_cluster": "clusterTwo", "leader_index": "testme"}'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "index_not_found_exception",
        "reason" : "no such index [testme]",
        "index_uuid" : "_na_",
        "index" : "testme"
      }
    ],
    "type" : "index_not_found_exception",
    "reason" : "no such index [testme]",
    "index_uuid" : "_na_",
    "index" : "testme"
  },
  "status" : 404
}

```

---

<div class="post-metadata">

**Author:** ![kyle\_che](https://avatars.discourse-cdn.com/v4/letter/k/ce73a5/32.png) [@kyle\_che](https://discuss.elastic.co/u/kyle_che)\
**Post date:** [November 11, 2020, 7:20pm UTC](https://discuss.elastic.co/t/remote-clusters-authentication/254988/20 "2020-11-11T19:20:15Z")

</div>

If i create the same index name on the remote cluster, i can then get it to create. The only problem is, the index does not replicate the data from the master to the slave. any ideas?

[Next page](https://discuss.elastic.co/t/remote-clusters-authentication/254988.md?page=2)