Remote clusters authentication

kyle_che · November 10, 2020, 10:59pm

I am trying to test out remote clusters and then cross cluster replication. When inputting information about the other cluster under "Remote Clusters", Connection is saying "Not connected". I'm guessing this is due to my clusters having passwords set. Is there a way to pass the username/password so that the connection status will say connected?

kyle_che · November 10, 2020, 11:00pm

This is what it currently looks like.

remote_clusters

warkolm · November 10, 2020, 11:03pm

The cluster trust isn't setup as a user/password approach. https://www.elastic.co/guide/en/elasticsearch/reference/current/cross-cluster-configuring.html explains it more.

What do your Elasticsearch logs show?

kyle_che · November 10, 2020, 11:24pm

[2020-11-10T22:11:04,656][WARN ][o.e.c.s.DiagnosticTrustManager] [d-gp2-kyleesdb1-1] failed to establish trust with server at [<unknown host>]; the server provided a certificate with subject name [CN=instance] and fingerprint [d215f08f8d81045fc228d607b5abe4c6e65b4b19]; the certificate does not have any subject alternative names; the certificate is issued by [CN=Elastic Certificate Tool Autogenerated CA]; the certificate is signed by (subject [CN=Elastic Certificate Tool Autogenerated CA] fingerprint [91ba3f59359417423c3442eda6bea3cb5e9af46c]) which is self-issued; the [CN=Elastic Certificate Tool Autogenerated CA] certificate is not trusted in this ssl context ([xpack.security.transport.ssl]); this ssl context does trust a certificate with subject [CN=Elastic Certificate Tool Autogenerated CA] but the trusted certificate has fingerprint [a1116e365cf3e89c41a907ce46c96585e9d9afb0]
sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:369) ~[?:?]
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:275) ~[?:?]
        at sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285) ~[?:?]
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144) ~[?:?]
        at org.elasticsearch.common.ssl.DiagnosticTrustManager.checkServerTrusted(DiagnosticTrustManager.java:110) [elasticsearch-ssl-config-7.9.3.jar:7.9.3]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1318) [?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1215) [?:?]
        at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1158) [?:?]
        at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) [?:?]
        at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) [?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1267) [?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1254) [?:?]
        at java.security.AccessController.doPrivileged(AccessController.java:691) [?:?]
        at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1199) [?:?]
        at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1542) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1556) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1440) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
        at java.lang.Thread.run(Thread.java:832) [?:?]
Caused by: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
        at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:158) ~[?:?]
        at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:84) ~[?:?]
        at java.security.cert.CertPathValidator.validate(CertPathValidator.java:309) ~[?:?]
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:364) ~[?:?]
        ... 37 more

kyle_che · November 10, 2020, 11:25pm

must both clusters have the same elastic-certificates.p12 file?

warkolm · November 10, 2020, 11:26pm

From that doc link;

Enable a trust relationship between the cluster used for performing cross cluster search (the local cluster) and all remote clusters. This can be done either by:

Using the same certificate authority to generate certificates for all connected clusters, or

Adding the CA certificate from the local cluster as a trusted CA in each remote cluster (see Transport TLS settings).

kyle_che · November 10, 2020, 11:35pm

that worked. one last question. I have "Remote Clusters" setup. I am missing the " Cross Cluster Replication" option on the left hand side. Does something have to be installed for this feature?

kyle_che · November 10, 2020, 11:39pm

looks like it's not free so it requires a license. thanks for your help.

warkolm · November 10, 2020, 11:41pm

You can upgrade a Basic license to a 30 day Platinum trial if you want - https://www.elastic.co/guide/en/kibana/current/managing-licenses.html.

kyle_che · November 10, 2020, 11:49pm

ok, did that and i'm getting the following error.

Can't create follower index

[index_not_found_exception] no such index [kibana_sample_data_flights], with { index_uuid="_na_" & index="kibana_sample_data_flights" }

* no such index [kibana_sample_data_flights]

warkolm · November 10, 2020, 11:50pm

Did you load the sample data? You can do that from the home screen - https://www.elastic.co/guide/en/kibana/current/tutorial-sample-data.html

kyle_che · November 10, 2020, 11:51pm

yes, it's been loaded but won't replicate.

# curl -XGET "elastic:***@localhost:9200/_cat/indices?v&index=kibana*&pretty"
health status index                        uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   kibana_sample_data_ecommerce gGFEHJKAREiFG2MueQ-2OA   1   1       4675            0      9.2mb          4.6mb
green  open   kibana_sample_data_logs      CLc7k_fNQoCJ5cYfpMPZpA   1   1      14074            0     19.1mb          9.6mb
green  open   kibana_sample_data_flights   nBDI26H1T4WNFXr5CGgwkg   1   1      13059            0     11.9mb            6mb

kyle_che · November 11, 2020, 12:14am

kibana error is:

{"type":"response","@timestamp":"2020-11-11T00:14:14Z","tags":[],"pid":15625,"method":"post","statusCode":404,"req":{"url":"/api/cross_cluster_replication/follower_indices","method":"post","headers":{"host":"d-gp2-kyleesdb1-1.imovetv.com:5601","connection":"keep-alive","content-length":"441","kbn-version":"7.9.3","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.183 Safari/537.36","content-type":"application/json","accept":"*/*","origin":"http://d-gp2-kyleesdb1-1.imovetv.com:5601","referer":"http://d-gp2-kyleesdb1-1.imovetv.com:5601/app/management/data/cross_cluster_replication/follower_indices/add","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9,es;q=0.8,mt;q=0.7"},"remoteAddress":"10.124.250.242","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.183 Safari/537.36","referer":"http://d-gp2-kyleesdb1-1.imovetv.com:5601/app/management/data/cross_cluster_replication/follower_indices/add"},"res":{"statusCode":404,"responseTime":142,"contentLength":9},"message":"POST /api/cross_cluster_replication/follower_indices 404 142ms - 9.0B"}

kyle_che · November 11, 2020, 12:42am

I have also created a new index testme with 1 doc and it gets the same error. What am i missng?

Can't create follower index
[index_not_found_exception] no such index [testme], with { index_uuid="_na_" & index="testme" }
no such index [testme]

warkolm · November 11, 2020, 12:48am

What the output from the GET /_ccr/stats endpoint?

kyle_che · November 11, 2020, 12:50am

here is what is the code that is being run...

curl -XGET "http://d-gp2-kyleesdb1-1.imovetv.com:9200/_cat/indices?v&index=test*&pretty"
health status index  uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   testme 8j-Sam_gS22BsqENTMMXzA   1   1          1            0      7.6kb          3.8kb

curl -XPUT "http://d-gp2-kyleesdb1-1.imovetv.com:9200/follow_testme/_ccr/follow" -H 'Content-Type: application/json' -d'{  "remote_cluster": "clusterTwo",  "leader_index": "testme",  "max_read_request_operation_count": 5120,  "max_outstanding_read_requests": 12,  "max_read_request_size": "32mb",  "max_write_request_operation_count": 5120,  "max_write_request_size": "9223372036854775807b",  "max_outstanding_write_requests": 9,  "max_write_buffer_count": 2147483647,  "max_write_buffer_size": "512mb",  "max_retry_delay": "500ms",  "read_poll_timeout": "1m"}'

{
  "error" : {
    "root_cause" : [
      {
        "type" : "index_not_found_exception",
        "reason" : "no such index [testme]",
        "index_uuid" : "_na_",
        "index" : "testme"
      }
    ],
    "type" : "index_not_found_exception",
    "reason" : "no such index [testme]",
    "index_uuid" : "_na_",
    "index" : "testme"
  },
  "status" : 404
}

kyle_che · November 11, 2020, 5:55pm

Here is the output from the command you sent me.

curl -XGET "http://d-gp2-kyleesdb1-1.imovetv.com:9200/_ccr/stats"
{
  "auto_follow_stats" : {
    "number_of_failed_follow_indices" : 0,
    "number_of_failed_remote_cluster_state_requests" : 0,
    "number_of_successful_follow_indices" : 0,
    "recent_auto_follow_errors" : [ ],
    "auto_followed_clusters" : [ ]
  },
  "follow_stats" : {
    "indices" : [ ]
  }
}

kyle_che · November 11, 2020, 6:03pm

just getting the below error.

kyle_che · November 11, 2020, 6:23pm

Tried to do it manually but same error.

curl -XPUT "elastic:***@d-gp2-kyleesdb1-1.imovetv.com:9200/testme-copy/_ccr/follow?pretty" -H 'Content-Type: application/json' -d'{  "remote_cluster": "clusterTwo",  "leader_index": "testme"}'
{
  "error" : {
    "root_cause" : [
      {
        "type" : "index_not_found_exception",
        "reason" : "no such index [testme]",
        "index_uuid" : "_na_",
        "index" : "testme"
      }
    ],
    "type" : "index_not_found_exception",
    "reason" : "no such index [testme]",
    "index_uuid" : "_na_",
    "index" : "testme"
  },
  "status" : 404
}

kyle_che · November 11, 2020, 7:20pm

If i create the same index name on the remote cluster, i can then get it to create. The only problem is, the index does not replicate the data from the master to the slave. any ideas?