Remote clusters authentication

I am trying to test out remote clusters and then cross cluster replication. When inputting information about the other cluster under "Remote Clusters", Connection is saying "Not connected". I'm guessing this is due to my clusters having passwords set. Is there a way to pass the username/password so that the connection status will say connected?

This is what it currently looks like.

remote_clusters

The cluster trust isn't setup as a user/password approach. https://www.elastic.co/guide/en/elasticsearch/reference/current/cross-cluster-configuring.html explains it more.

What do your Elasticsearch logs show?

[2020-11-10T22:11:04,656][WARN ][o.e.c.s.DiagnosticTrustManager] [d-gp2-kyleesdb1-1] failed to establish trust with server at [<unknown host>]; the server provided a certificate with subject name [CN=instance] and fingerprint [d215f08f8d81045fc228d607b5abe4c6e65b4b19]; the certificate does not have any subject alternative names; the certificate is issued by [CN=Elastic Certificate Tool Autogenerated CA]; the certificate is signed by (subject [CN=Elastic Certificate Tool Autogenerated CA] fingerprint [91ba3f59359417423c3442eda6bea3cb5e9af46c]) which is self-issued; the [CN=Elastic Certificate Tool Autogenerated CA] certificate is not trusted in this ssl context ([xpack.security.transport.ssl]); this ssl context does trust a certificate with subject [CN=Elastic Certificate Tool Autogenerated CA] but the trusted certificate has fingerprint [a1116e365cf3e89c41a907ce46c96585e9d9afb0]
sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:369) ~[?:?]
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:275) ~[?:?]
        at sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144) ~[?:?]
        at org.elasticsearch.common.ssl.DiagnosticTrustManager.checkServerTrusted(DiagnosticTrustManager.java:110) [elasticsearch-ssl-config-7.9.3.jar:7.9.3]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1318) [?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1215) [?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1158) [?:?]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) [?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) [?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1267) [?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1254) [?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:691) [?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1199) [?:?]
        at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1542) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1556) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1440) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
        at java.lang.Thread.run(Thread.java:832) [?:?]
Caused by: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:158) ~[?:?]
        at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:84) ~[?:?]
        at java.security.cert.CertPathValidator.validate(CertPathValidator.java:309) ~[?:?]
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:364) ~[?:?]
        ... 37 more

must both clusters have the same elastic-certificates.p12 file?

From that doc link;

Enable a trust relationship between the cluster used for performing cross cluster search (the local cluster) and all remote clusters. This can be done either by:

  • Using the same certificate authority to generate certificates for all connected clusters, or
  • Adding the CA certificate from the local cluster as a trusted CA in each remote cluster (see Transport TLS settings).

that worked. one last question. I have "Remote Clusters" setup. I am missing the " Cross Cluster Replication" option on the left hand side. Does something have to be installed for this feature?

looks like it's not free so it requires a license. thanks for your help.

You can upgrade a Basic license to a 30 day Platinum trial if you want - https://www.elastic.co/guide/en/kibana/current/managing-licenses.html.

ok, did that and i'm getting the following error.

Can't create follower index

[index_not_found_exception] no such index [kibana_sample_data_flights], with { index_uuid="_na_" & index="kibana_sample_data_flights" }

* no such index [kibana_sample_data_flights]

Did you load the sample data? You can do that from the home screen - https://www.elastic.co/guide/en/kibana/current/tutorial-sample-data.html

yes, it's been loaded but won't replicate.

# curl -XGET "elastic:***@localhost:9200/_cat/indices?v&index=kibana*&pretty"
health status index                        uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   kibana_sample_data_ecommerce gGFEHJKAREiFG2MueQ-2OA   1   1       4675            0      9.2mb          4.6mb
green  open   kibana_sample_data_logs      CLc7k_fNQoCJ5cYfpMPZpA   1   1      14074            0     19.1mb          9.6mb
green  open   kibana_sample_data_flights   nBDI26H1T4WNFXr5CGgwkg   1   1      13059            0     11.9mb            6mb

kibana error is:

{"type":"response","@timestamp":"2020-11-11T00:14:14Z","tags":[],"pid":15625,"method":"post","statusCode":404,"req":{"url":"/api/cross_cluster_replication/follower_indices","method":"post","headers":{"host":"d-gp2-kyleesdb1-1.imovetv.com:5601","connection":"keep-alive","content-length":"441","kbn-version":"7.9.3","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.183 Safari/537.36","content-type":"application/json","accept":"*/*","origin":"http://d-gp2-kyleesdb1-1.imovetv.com:5601","referer":"http://d-gp2-kyleesdb1-1.imovetv.com:5601/app/management/data/cross_cluster_replication/follower_indices/add","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9,es;q=0.8,mt;q=0.7"},"remoteAddress":"10.124.250.242","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.183 Safari/537.36","referer":"http://d-gp2-kyleesdb1-1.imovetv.com:5601/app/management/data/cross_cluster_replication/follower_indices/add"},"res":{"statusCode":404,"responseTime":142,"contentLength":9},"message":"POST /api/cross_cluster_replication/follower_indices 404 142ms - 9.0B"}

I have also created a new index testme with 1 doc and it gets the same error. What am i missng?

Can't create follower index
[index_not_found_exception] no such index [testme], with { index_uuid="_na_" & index="testme" }
no such index [testme]

What the output from the GET /_ccr/stats endpoint?

here is what is the code that is being run...

curl -XGET "http://d-gp2-kyleesdb1-1.imovetv.com:9200/_cat/indices?v&index=test*&pretty"
health status index  uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   testme 8j-Sam_gS22BsqENTMMXzA   1   1          1            0      7.6kb          3.8kb

curl -XPUT "http://d-gp2-kyleesdb1-1.imovetv.com:9200/follow_testme/_ccr/follow" -H 'Content-Type: application/json' -d'{  "remote_cluster": "clusterTwo",  "leader_index": "testme",  "max_read_request_operation_count": 5120,  "max_outstanding_read_requests": 12,  "max_read_request_size": "32mb",  "max_write_request_operation_count": 5120,  "max_write_request_size": "9223372036854775807b",  "max_outstanding_write_requests": 9,  "max_write_buffer_count": 2147483647,  "max_write_buffer_size": "512mb",  "max_retry_delay": "500ms",  "read_poll_timeout": "1m"}'

{
  "error" : {
    "root_cause" : [
      {
        "type" : "index_not_found_exception",
        "reason" : "no such index [testme]",
        "index_uuid" : "_na_",
        "index" : "testme"
      }
    ],
    "type" : "index_not_found_exception",
    "reason" : "no such index [testme]",
    "index_uuid" : "_na_",
    "index" : "testme"
  },
  "status" : 404
}

Here is the output from the command you sent me.

curl -XGET "http://d-gp2-kyleesdb1-1.imovetv.com:9200/_ccr/stats"
{
  "auto_follow_stats" : {
    "number_of_failed_follow_indices" : 0,
    "number_of_failed_remote_cluster_state_requests" : 0,
    "number_of_successful_follow_indices" : 0,
    "recent_auto_follow_errors" : [ ],
    "auto_followed_clusters" : [ ]
  },
  "follow_stats" : {
    "indices" : [ ]
  }
}

just getting the below error.

Tried to do it manually but same error.

curl -XPUT "elastic:***@d-gp2-kyleesdb1-1.imovetv.com:9200/testme-copy/_ccr/follow?pretty" -H 'Content-Type: application/json' -d'{  "remote_cluster": "clusterTwo",  "leader_index": "testme"}'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "index_not_found_exception",
        "reason" : "no such index [testme]",
        "index_uuid" : "_na_",
        "index" : "testme"
      }
    ],
    "type" : "index_not_found_exception",
    "reason" : "no such index [testme]",
    "index_uuid" : "_na_",
    "index" : "testme"
  },
  "status" : 404
}

If i create the same index name on the remote cluster, i can then get it to create. The only problem is, the index does not replicate the data from the master to the slave. any ideas?