# Elasticsearch Cross Cluster Search Implementation **URL:** \ **Category:** Elasticsearch\ **Tags:** elastic-stack-security, ccs-cross-cluster-search\ **Created:** [December 28, 2020, 5:54am UTC](https://discuss.elastic.co/t/elasticsearch-cross-cluster-search-implementation/259708 "2020-12-28T05:54:49Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![tusharnemade](https://avatars.discourse-cdn.com/v4/letter/t/67e7ee/32.png) [@tusharnemade](https://discuss.elastic.co/u/tusharnemade)\ **Post date:** [December 28, 2020, 5:54am UTC](https://discuss.elastic.co/t/elasticsearch-cross-cluster-search-implementation/259708/1 "2020-12-28T05:54:49Z")
Hello Team I am using Elasticsearch version 7.8.0. I am having two clusters : a. Local cluster b. remote cluster. Remote Cluster has Authentication implemented with CA Certificates generated for himself. Similarly Local Cluster has Authentication implemented with CA Certificates generated for himself. In Local Cluster, elasticsearch.yl file has below entires -- ``` xpack.license.self_generated.type: basic xpack.security.enabled: true xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: certificate xpack.security.transport.ssl.keystore.path: /mnt/elasticsearch-7.8.0/config/certs/elastic-certificates.p12 xpack.security.transport.ssl.truststore.path: /mnt/elasticsearch-7.8.0/config/certs/elastic-certificates.p12 cluster.remote.cluster_rmt.seeds: 172.22.11.74:9300 cluster.remote.cluster_rmt.transport.ping_schedule: 60s ``` Remote Cluster has ```auto xpack.license.self_generated.type: basic xpack.security.enabled: true xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.verification_mode: certificate xpack.security.transport.ssl.keystore.path: /mnt/elasticsearch-7.8.0/config/certs/elastic-certificates.p12 xpack.security.transport.ssl.truststore.path: /mnt/elasticsearch-7.8.0/config/certs/elastic-certificates.p12 xpack.security.transport.ssl.client_authentication: required ``` Note : Both Cluster has independent CA Certificates generated using "elasticsearch-certutil" utility. When i start Local Cluster Node , I am getting below Error : `[2020-12-28T11:10:12,634][WARN][o.e.c.s.DiagnosticTrustManager] [node-1] failed to establish trust with server at []; the server provided a certificate with subject name [CN=instance] and fingerprint [275fc1cae0eb692423faa56aecbb8056dbbf6308]; the certificate has subject alternative names [IP:172.22.11.74,IP:172.22.11.73]; the certificate is issued by [CN=Elastic Certificate Tool Autogenerated CA]; the certificate is signed by (subject [CN=Elastic Certificate Tool Autogenerated CA] fingerprint [eb9e4643caef03ccdc17bc869615d50ad4c37b85]) which is self-issued; the [CN=Elastic Certificate Tool Autogenerated CA] certificate is not trusted in this ssl context ([xpack.security.transport.ssl]); this ssl context does trust a certificate with subject [CN=Elastic Certificate Tool Autogenerated CA] but the trusted certificate has fingerprint [91686e888ea31302ef38e798a538f7eff623da58]` And at same time in remote cluster it has below message `[2020-12-28T11:09:55,953][WARN][o.e.x.c.s.t.n.SecurityNetty4Transport] [node-1] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=0.0.0.0/0.0.0.0:9300, remoteAddress=/172.22.11.73:36856}` Could you please guide me to fix this error. Thanks Tushar Nemade ---
**Author:** ![tusharnemade](https://avatars.discourse-cdn.com/v4/letter/t/67e7ee/32.png) [@tusharnemade](https://discuss.elastic.co/u/tusharnemade)\ **Post date:** [December 28, 2020, 5:58am UTC](https://discuss.elastic.co/t/elasticsearch-cross-cluster-search-implementation/259708/2 "2020-12-28T05:58:27Z")
> [@tusharnemade](#): > > `xpack.security.transport.ssl.verification_mode:` I have tried using xpack.security.transport.ssl.verification\_mode: full But it did not helped me. ---
**Author:** ![tusharnemade](https://avatars.discourse-cdn.com/v4/letter/t/67e7ee/32.png) [@tusharnemade](https://discuss.elastic.co/u/tusharnemade)\ **Post date:** [December 28, 2020, 8:39am UTC](https://discuss.elastic.co/t/elasticsearch-cross-cluster-search-implementation/259708/3 "2020-12-28T08:39:57Z")
can someone please help me ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [December 28, 2020, 9:10am UTC](https://discuss.elastic.co/t/elasticsearch-cross-cluster-search-implementation/259708/4 "2020-12-28T09:10:43Z")
Please be patient in waiting for responses to your question and refrain from pinging multiple times asking for a response or opening multiple topics for the same question. This is a community forum, it may take time for someone to reply to your question. For more information please refer to the [Community Code of Conduct](https://www.elastic.co/community/codeofconduct) specifically the section "Be patient". Also, please refrain from pinging folks directly, this is a forum and anyone that participates might be able to assist you. If you are in need of a service with an SLA that covers response times for questions then you may want to consider talking to us about a [subscription](https://www.elastic.co/subscriptions). It's fine to answer on your own thread after 2 or 3 days (not including weekends) if you don't have an answer. ---
**Author:** ![tusharnemade](https://avatars.discourse-cdn.com/v4/letter/t/67e7ee/32.png) [@tusharnemade](https://discuss.elastic.co/u/tusharnemade)\ **Post date:** [December 28, 2020, 9:53am UTC](https://discuss.elastic.co/t/elasticsearch-cross-cluster-search-implementation/259708/5 "2020-12-28T09:53:28Z")
Hello David Please excuse me , if my actions indicated that I am being impatient and pushing in this open-forum. I did not intended the same. Thanks Tushar Nemade ---
**Author:** ![BenB196](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benb196/32/83401_2.png) [@BenB196](https://discuss.elastic.co/u/BenB196)\ **Post date:** [December 28, 2020, 3:37pm UTC](https://discuss.elastic.co/t/elasticsearch-cross-cluster-search-implementation/259708/6 "2020-12-28T15:37:33Z")
@tusharnemade you probably have two (2) options here: 1. Set `xpack.security.transport.ssl.verification_mode: false` note that this would disable ssl verification entirely, which is not the best idea on the transport layer. - Setting `xpack.security.transport.ssl.verification_mode: full` didn't do anything as `full` is the default and stricter than certificate, and your error is related to the certificate not being trusted. 2. It looks like you generated the certs via Elasticsearch cert tool. Therefore, each server (local and remote) should've generated a CA (certificate authority) file. If you take the public part of those CA's and install/add them as trusted root CA's on all servers, this should fix the issue. - You should do this to the remote first, then to the local. - I don't know if Elasticsearch requires a restart after doing this. - This may not work, and you may need to import the CA's into Java's keystore directly via keytool - I didn't add instructions here on purpose as its highly dependent on your OS (operating system). I recommend you search for the instructions appropriate to your OS. ---
**Author:** ![tusharnemade](https://avatars.discourse-cdn.com/v4/letter/t/67e7ee/32.png) [@tusharnemade](https://discuss.elastic.co/u/tusharnemade)\ **Post date:** [December 29, 2020, 4:48am UTC](https://discuss.elastic.co/t/elasticsearch-cross-cluster-search-implementation/259708/7 "2020-12-29T04:48:45Z")
Hello Ben Thanks For your response. Option 1 , is not possible for me as it will void my usage of SSL. Option 2, I will checkout the options you have specified. Although as per documentation of Elasticsearch `https://www.elastic.co/guide/en/elasticsearch/reference/master/cross-cluster-configuring.html` , it does not mention about any of these to be carried out. It just states a statement that `Adding the CA certificate from the local cluster as a trusted CA in each remote cluster (see Transport TLS/SSL settings).` I am trying to understan what this TLS/SSL ( specifically which one) will help me in getting this done. Thanks Tushar Nemade ---
**Author:** ![tusharnemade](https://avatars.discourse-cdn.com/v4/letter/t/67e7ee/32.png) [@tusharnemade](https://discuss.elastic.co/u/tusharnemade)\ **Post date:** [December 29, 2020, 12:29pm UTC](https://discuss.elastic.co/t/elasticsearch-cross-cluster-search-implementation/259708/8 "2020-12-29T12:29:33Z")
Hi Ben What I did today is , I created CA Certificate { elastic-stack-ca.p12 } from Local Cluster node. Then generated each local node certificate { elastic-certificates.p12 } by using command :-- ```auto elasticsearch-certutil cert --ca elastic-stack-ca.p12 --ip , -v ``` I copied , CA certificate and Node Certificate on each node. Then my Cluster was working fine as expected for cross-cluster search purpose. So as per documentation : ```auto a) Using the same certificate authority to generate certificates for all connected clusters, or b) Adding the CA certificate from the local cluster as a trusted CA in each remote cluster. ``` I used Method a). As for method b) -- I as not getting any solution to be how it should be implemented. I will wait in her for few days , if i receive any response on b) method of implementation. As for me production implementation has some time-window left yet. Thanks Tushar Nemade ---
**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\ **Post date:** [December 30, 2020, 5:56am UTC](https://discuss.elastic.co/t/elasticsearch-cross-cluster-search-implementation/259708/9 "2020-12-30T05:56:48Z")
You can copy trusted CAs between truststores using Java's `keytool` You will need to run this twice, so that you can copy each CA from one cluster to the other. ```auto keytool -importkeystore \ -srckeystore /path/to/other-cluster/elastic-certificates.p12 -srcstorepass "" -srcalias "ca" \ -destkeystore /path/to/this-cluster/elastic-certificates.p12 -deststorepass "" -destalias "remo te-ca" ``` ---
**Author:** ![tusharnemade](https://avatars.discourse-cdn.com/v4/letter/t/67e7ee/32.png) [@tusharnemade](https://discuss.elastic.co/u/tusharnemade)\ **Post date:** [December 30, 2020, 6:34am UTC](https://discuss.elastic.co/t/elasticsearch-cross-cluster-search-implementation/259708/10 "2020-12-30T06:34:45Z")
Hi Tim Thanks for your response. I will execute now this and confirm in here if it worked or not. ---
**Author:** ![tusharnemade](https://avatars.discourse-cdn.com/v4/letter/t/67e7ee/32.png) [@tusharnemade](https://discuss.elastic.co/u/tusharnemade)\ **Post date:** [December 30, 2020, 12:36pm UTC](https://discuss.elastic.co/t/elasticsearch-cross-cluster-search-implementation/259708/11 "2020-12-30T12:36:57Z")
Hi Tim I did the execution of `keytool` as you mentioned in your post. ```auto keytool -importkeystore -srckeystore /mnt/elasticsearch-7.8.0/config/certs/elastic-certificates-elk-localnode-1.p12 -srcstorepass "elastic" -destkeystore /mnt/elasticsearch-7.8.0/config/certs/elastic-certificates.p12 -deststorepass "elastic" Importing keystore /mnt/elasticsearch-7.8.0/config/certs/elastic-certificates-elk-localnode-1.p12 to /mnt/elasticsearch-7.8.0/config/certs/elastic-certificates.p12... Existing entry alias instance exists, overwrite? [no]: no Enter new alias name (RETURN to cancel import for this entry): elk-localnode-instance Entry for alias instance successfully imported. Existing entry alias ca exists, overwrite? [no]: no Enter new alias name (RETURN to cancel import for this entry): elk-localnode-ca Entry for alias ca successfully imported. Import command completed: 2 entries successfully imported, 0 entries failed or cancelled ``` After importing , I checked the keystore contents , it is showing me both server details , as expected. But still I received an error message of :-- ```auto [WARN][o.e.x.c.s.t.n.SecurityNetty4Transport] [node-1] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=0.0.0.0/0.0.0.0:9300, remoteAddress=/172.22.11.73:38922} ``` I did only importing of CA certificate from Local Cluster Node to Remote Cluster Node. As this is my use case and required as per elasticsearch documentation. Could you please help me in understanding ... will different "ALIAS" names in Local Cluster Node Certificate and Remote Cluster Node Certificate will cause an issue ? OR There is some other reason for this error ... Thanks Tushar Nemade ---
**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\ **Post date:** [December 30, 2020, 10:22pm UTC](https://discuss.elastic.co/t/elasticsearch-cross-cluster-search-implementation/259708/12 "2020-12-30T22:22:37Z")
> [@tusharnemade](#): > > I did only importing of CA certificate from Local Cluster Node to Remote Cluster Node. As this is my use case and required as per elasticsearch documentation. You need to establish trust on both clusters. The documentation is misleading here, I will get it fixed up. ---
**Author:** ![tusharnemade](https://avatars.discourse-cdn.com/v4/letter/t/67e7ee/32.png) [@tusharnemade](https://discuss.elastic.co/u/tusharnemade)\ **Post date:** [December 31, 2020, 3:13am UTC](https://discuss.elastic.co/t/elasticsearch-cross-cluster-search-implementation/259708/13 "2020-12-31T03:13:51Z")
Hi Tim Thanks for your response. I will perform this from both side and update in here my results. ---
**Author:** ![tusharnemade](https://avatars.discourse-cdn.com/v4/letter/t/67e7ee/32.png) [@tusharnemade](https://discuss.elastic.co/u/tusharnemade)\ **Post date:** [December 31, 2020, 7:13am UTC](https://discuss.elastic.co/t/elasticsearch-cross-cluster-search-implementation/259708/14 "2020-12-31T07:13:04Z")
Hi Tim I have executed keytool command in following ways to have both nodes of different cluster each , have their CA certificates trusted. On Local Cluster node ```auto [esearch@elk-localnode-1 ~]$ keytool -importkeystore -srckeystore /mnt/elasticsearch-7.8.0/config/certs/elastic-certificates-elk-remotenode-1.p12 -srcstorepass "elastic" -destkeystore /mnt/elasticsearch-7.8.0/config/certs/elastic-certificates.p12 -deststorepass "elastic" Importing keystore /mnt/elasticsearch-7.8.0/config/certs/elastic-certificates-elk-remotenode-1.p12 to /mnt/elasticsearch-7.8.0/config/certs/elastic-certificates.p12... Existing entry alias instance exists, overwrite? [no]: no Enter new alias name (RETURN to cancel import for this entry): elk-remotenode-1-remote Entry for alias instance successfully imported. Existing entry alias ca exists, overwrite? [no]: no Enter new alias name (RETURN to cancel import for this entry): elk-remotenode-1-ca Entry for alias ca successfully imported. Import command completed: 2 entries successfully imported, 0 entries failed or cancelled [esearch@elk-localnode-1 ~]$ ``` On Remote Cluster Node ```auto [esearch@elk-remotenode-1 ~]$ keytool -importkeystore -srckeystore /mnt/elasticsearch-7.8.0/config/certs/elastic-certificates-elk-localnode-1.p12 -srcstorepass "elastic" -destkeystore /mnt/elasticsearch-7.8.0/config/certs/elastic-certificates.p12 -deststorepass "elastic" Importing keystore /mnt/elasticsearch-7.8.0/config/certs/elastic-certificates-elk-localnode-1.p12 to /mnt/elasticsearch-7.8.0/config/certs/elastic-certificates.p12... Existing entry alias instance exists, overwrite? [no]: no Enter new alias name (RETURN to cancel import for this entry): elk-localnode-instance Entry for alias instance successfully imported. Existing entry alias ca exists, overwrite? [no]: no Enter new alias name (RETURN to cancel import for this entry): elk-localnode-ca Entry for alias ca successfully imported. Import command completed: 2 entries successfully imported, 0 entries failed or cancelled ``` It did worked successfully as expected. I will accept it as a solution in this forum. Could you please help me in one more thing ... I am generating Nodes Certificates using ```auto elasticsearch-certutil ca ``` How to pass Alias names for Instance and CA while creating certificates. Thanks Tushar Nemade ---
**Author:** ![tusharnemade](https://avatars.discourse-cdn.com/v4/letter/t/67e7ee/32.png) [@tusharnemade](https://discuss.elastic.co/u/tusharnemade)\ **Post date:** [December 31, 2020, 8:44am UTC](https://discuss.elastic.co/t/elasticsearch-cross-cluster-search-implementation/259708/15 "2020-12-31T08:44:28Z")
Could you please help me in one more thing ... I am generating Nodes Certificates using ```auto elasticsearch-certutil ca ``` How to pass Alias names for Instance and CA while creating certificates. Thanks Tushar Nemade ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 28, 2021, 8:44am UTC](https://discuss.elastic.co/t/elasticsearch-cross-cluster-search-implementation/259708/16 "2021-01-28T08:44:36Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.