I am trying to test out remote clusters and then cross cluster replication. When inputting information about the other cluster under "Remote Clusters", Connection is saying "Not connected". I'm guessing this is due to my clusters having passwords set. Is there a way to pass the username/password so that the connection status will say connected?
This is what it currently looks like.
The cluster trust isn't setup as a user/password approach. https://www.elastic.co/guide/en/elasticsearch/reference/current/cross-cluster-configuring.html explains it more.
What do your Elasticsearch logs show?
[2020-11-10T22:11:04,656][WARN ][o.e.c.s.DiagnosticTrustManager] [d-gp2-kyleesdb1-1] failed to establish trust with server at [<unknown host>]; the server provided a certificate with subject name [CN=instance] and fingerprint [d215f08f8d81045fc228d607b5abe4c6e65b4b19]; the certificate does not have any subject alternative names; the certificate is issued by [CN=Elastic Certificate Tool Autogenerated CA]; the certificate is signed by (subject [CN=Elastic Certificate Tool Autogenerated CA] fingerprint [91ba3f59359417423c3442eda6bea3cb5e9af46c]) which is self-issued; the [CN=Elastic Certificate Tool Autogenerated CA] certificate is not trusted in this ssl context ([xpack.security.transport.ssl]); this ssl context does trust a certificate with subject [CN=Elastic Certificate Tool Autogenerated CA] but the trusted certificate has fingerprint [a1116e365cf3e89c41a907ce46c96585e9d9afb0]
sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:369) ~[?:?]
at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:275) ~[?:?]
at sun.security.validator.Validator.validate(Validator.java:264) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:285) ~[?:?]
at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144) ~[?:?]
at org.elasticsearch.common.ssl.DiagnosticTrustManager.checkServerTrusted(DiagnosticTrustManager.java:110) [elasticsearch-ssl-config-7.9.3.jar:7.9.3]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1318) [?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1215) [?:?]
at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1158) [?:?]
at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) [?:?]
at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) [?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1267) [?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1254) [?:?]
at java.security.AccessController.doPrivileged(AccessController.java:691) [?:?]
at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1199) [?:?]
at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1542) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1556) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1440) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) [netty-handler-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) [netty-codec-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at java.lang.Thread.run(Thread.java:832) [?:?]
Caused by: java.security.cert.CertPathValidatorException: Path does not chain with any of the trust anchors
at sun.security.provider.certpath.PKIXCertPathValidator.validate(PKIXCertPathValidator.java:158) ~[?:?]
at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:84) ~[?:?]
at java.security.cert.CertPathValidator.validate(CertPathValidator.java:309) ~[?:?]
at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:364) ~[?:?]
... 37 more
must both clusters have the same elastic-certificates.p12 file?
From that doc link;
Enable a trust relationship between the cluster used for performing cross cluster search (the local cluster) and all remote clusters. This can be done either by:
- Using the same certificate authority to generate certificates for all connected clusters, or
- Adding the CA certificate from the local cluster as a trusted CA in each remote cluster (see Transport TLS settings).
that worked. one last question. I have "Remote Clusters" setup. I am missing the " Cross Cluster Replication" option on the left hand side. Does something have to be installed for this feature?
looks like it's not free so it requires a license. thanks for your help.
You can upgrade a Basic license to a 30 day Platinum trial if you want - https://www.elastic.co/guide/en/kibana/current/managing-licenses.html.
ok, did that and i'm getting the following error.
Can't create follower index
[index_not_found_exception] no such index [kibana_sample_data_flights], with { index_uuid="_na_" & index="kibana_sample_data_flights" }
* no such index [kibana_sample_data_flights]
Did you load the sample data? You can do that from the home screen - https://www.elastic.co/guide/en/kibana/current/tutorial-sample-data.html
yes, it's been loaded but won't replicate.
# curl -XGET "elastic:***@localhost:9200/_cat/indices?v&index=kibana*&pretty"
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open kibana_sample_data_ecommerce gGFEHJKAREiFG2MueQ-2OA 1 1 4675 0 9.2mb 4.6mb
green open kibana_sample_data_logs CLc7k_fNQoCJ5cYfpMPZpA 1 1 14074 0 19.1mb 9.6mb
green open kibana_sample_data_flights nBDI26H1T4WNFXr5CGgwkg 1 1 13059 0 11.9mb 6mb
kibana error is:
{"type":"response","@timestamp":"2020-11-11T00:14:14Z","tags":[],"pid":15625,"method":"post","statusCode":404,"req":{"url":"/api/cross_cluster_replication/follower_indices","method":"post","headers":{"host":"d-gp2-kyleesdb1-1.imovetv.com:5601","connection":"keep-alive","content-length":"441","kbn-version":"7.9.3","user-agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.183 Safari/537.36","content-type":"application/json","accept":"*/*","origin":"http://d-gp2-kyleesdb1-1.imovetv.com:5601","referer":"http://d-gp2-kyleesdb1-1.imovetv.com:5601/app/management/data/cross_cluster_replication/follower_indices/add","accept-encoding":"gzip, deflate","accept-language":"en-US,en;q=0.9,es;q=0.8,mt;q=0.7"},"remoteAddress":"10.124.250.242","userAgent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.183 Safari/537.36","referer":"http://d-gp2-kyleesdb1-1.imovetv.com:5601/app/management/data/cross_cluster_replication/follower_indices/add"},"res":{"statusCode":404,"responseTime":142,"contentLength":9},"message":"POST /api/cross_cluster_replication/follower_indices 404 142ms - 9.0B"}
I have also created a new index testme with 1 doc and it gets the same error. What am i missng?
Can't create follower index
[index_not_found_exception] no such index [testme], with { index_uuid="_na_" & index="testme" }
no such index [testme]
What the output from the GET /_ccr/stats
endpoint?
here is what is the code that is being run...
curl -XGET "http://d-gp2-kyleesdb1-1.imovetv.com:9200/_cat/indices?v&index=test*&pretty"
health status index uuid pri rep docs.count docs.deleted store.size pri.store.size
green open testme 8j-Sam_gS22BsqENTMMXzA 1 1 1 0 7.6kb 3.8kb
curl -XPUT "http://d-gp2-kyleesdb1-1.imovetv.com:9200/follow_testme/_ccr/follow" -H 'Content-Type: application/json' -d'{ "remote_cluster": "clusterTwo", "leader_index": "testme", "max_read_request_operation_count": 5120, "max_outstanding_read_requests": 12, "max_read_request_size": "32mb", "max_write_request_operation_count": 5120, "max_write_request_size": "9223372036854775807b", "max_outstanding_write_requests": 9, "max_write_buffer_count": 2147483647, "max_write_buffer_size": "512mb", "max_retry_delay": "500ms", "read_poll_timeout": "1m"}'
{
"error" : {
"root_cause" : [
{
"type" : "index_not_found_exception",
"reason" : "no such index [testme]",
"index_uuid" : "_na_",
"index" : "testme"
}
],
"type" : "index_not_found_exception",
"reason" : "no such index [testme]",
"index_uuid" : "_na_",
"index" : "testme"
},
"status" : 404
}
Here is the output from the command you sent me.
curl -XGET "http://d-gp2-kyleesdb1-1.imovetv.com:9200/_ccr/stats"
{
"auto_follow_stats" : {
"number_of_failed_follow_indices" : 0,
"number_of_failed_remote_cluster_state_requests" : 0,
"number_of_successful_follow_indices" : 0,
"recent_auto_follow_errors" : [ ],
"auto_followed_clusters" : [ ]
},
"follow_stats" : {
"indices" : [ ]
}
}
Tried to do it manually but same error.
curl -XPUT "elastic:***@d-gp2-kyleesdb1-1.imovetv.com:9200/testme-copy/_ccr/follow?pretty" -H 'Content-Type: application/json' -d'{ "remote_cluster": "clusterTwo", "leader_index": "testme"}'
{
"error" : {
"root_cause" : [
{
"type" : "index_not_found_exception",
"reason" : "no such index [testme]",
"index_uuid" : "_na_",
"index" : "testme"
}
],
"type" : "index_not_found_exception",
"reason" : "no such index [testme]",
"index_uuid" : "_na_",
"index" : "testme"
},
"status" : 404
}
If i create the same index name on the remote cluster, i can then get it to create. The only problem is, the index does not replicate the data from the master to the slave. any ideas?