Elasticsearch Cross Cluster Search Implementation

tusharnemade · December 28, 2020, 5:54am

Hello Team

I am using Elasticsearch version 7.8.0.

I am having two clusters :

a. Local cluster
b. remote cluster.

Remote Cluster has Authentication implemented with CA Certificates generated for himself.

Similarly Local Cluster has Authentication implemented with CA Certificates generated for himself.

In Local Cluster, elasticsearch.yl file has below entires --

    xpack.license.self_generated.type: basic
    xpack.security.enabled: true
    xpack.security.transport.ssl.enabled: true
    xpack.security.transport.ssl.verification_mode: certificate
    xpack.security.transport.ssl.keystore.path: /mnt/elasticsearch-7.8.0/config/certs/elastic-certificates.p12
    xpack.security.transport.ssl.truststore.path: /mnt/elasticsearch-7.8.0/config/certs/elastic-certificates.p12

    cluster.remote.cluster_rmt.seeds: 172.22.11.74:9300
    cluster.remote.cluster_rmt.transport.ping_schedule: 60s

Remote Cluster has

xpack.license.self_generated.type: basic
xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /mnt/elasticsearch-7.8.0/config/certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: /mnt/elasticsearch-7.8.0/config/certs/elastic-certificates.p12
xpack.security.transport.ssl.client_authentication: required

Note : Both Cluster has independent CA Certificates generated using "elasticsearch-certutil" utility.

When i start Local Cluster Node , I am getting below Error :

[2020-12-28T11:10:12,634][WARN ][o.e.c.s.DiagnosticTrustManager] [node-1] failed to establish trust with server at [<unknown host>]; the server provided a certificate with subject name [CN=instance] and fingerprint [275fc1cae0eb692423faa56aecbb8056dbbf6308]; the certificate has subject alternative names [IP:172.22.11.74,IP:172.22.11.73]; the certificate is issued by [CN=Elastic Certificate Tool Autogenerated CA]; the certificate is signed by (subject [CN=Elastic Certificate Tool Autogenerated CA] fingerprint [eb9e4643caef03ccdc17bc869615d50ad4c37b85]) which is self-issued; the [CN=Elastic Certificate Tool Autogenerated CA] certificate is not trusted in this ssl context ([xpack.security.transport.ssl]); this ssl context does trust a certificate with subject [CN=Elastic Certificate Tool Autogenerated CA] but the trusted certificate has fingerprint [91686e888ea31302ef38e798a538f7eff623da58]

And at same time in remote cluster it has below message

[2020-12-28T11:09:55,953][WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [node-1] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=0.0.0.0/0.0.0.0:9300, remoteAddress=/172.22.11.73:36856}

Could you please guide me to fix this error.

Thanks
Tushar Nemade

tusharnemade · December 28, 2020, 5:58am

I have tried using

xpack.security.transport.ssl.verification_mode: full

But it did not helped me.

tusharnemade · December 28, 2020, 8:39am

can someone please help me

dadoonet · December 28, 2020, 9:10am

Please be patient in waiting for responses to your question and refrain from pinging multiple times asking for a response or opening multiple topics for the same question. This is a community forum, it may take time for someone to reply to your question. For more information please refer to the Community Code of Conduct specifically the section "Be patient". Also, please refrain from pinging folks directly, this is a forum and anyone that participates might be able to assist you.

If you are in need of a service with an SLA that covers response times for questions then you may want to consider talking to us about a subscription.

It's fine to answer on your own thread after 2 or 3 days (not including weekends) if you don't have an answer.

tusharnemade · December 28, 2020, 9:53am

Hello David

Please excuse me , if my actions indicated that I am being impatient and pushing in this open-forum. I did not intended the same.

Thanks
Tushar Nemade

BenB196 · December 28, 2020, 3:37pm

@tusharnemade you probably have two (2) options here:

Set xpack.security.transport.ssl.verification_mode: false note that this would disable ssl verification entirely, which is not the best idea on the transport layer.
- Setting xpack.security.transport.ssl.verification_mode: full didn't do anything as full is the default and stricter than certificate, and your error is related to the certificate not being trusted.
It looks like you generated the certs via Elasticsearch cert tool. Therefore, each server (local and remote) should've generated a CA (certificate authority) file. If you take the public part of those CA's and install/add them as trusted root CA's on all servers, this should fix the issue.
- You should do this to the remote first, then to the local.
- I don't know if Elasticsearch requires a restart after doing this.
- This may not work, and you may need to import the CA's into Java's keystore directly via keytool
- I didn't add instructions here on purpose as its highly dependent on your OS (operating system). I recommend you search for the instructions appropriate to your OS.

tusharnemade · December 29, 2020, 4:48am

Hello Ben
Thanks For your response.

Option 1 , is not possible for me as it will void my usage of SSL.

Option 2, I will checkout the options you have specified. Although as per documentation of Elasticsearch

https://www.elastic.co/guide/en/elasticsearch/reference/master/cross-cluster-configuring.html

, it does not mention about any of these to be carried out. It just states a statement that

Adding the CA certificate from the local cluster as a trusted CA in each remote cluster (see Transport TLS/SSL settings).
I am trying to understan what this TLS/SSL ( specifically which one) will help me in getting this done.

Thanks
Tushar Nemade

tusharnemade · December 29, 2020, 12:29pm

Hi Ben

What I did today is , I created CA Certificate { elastic-stack-ca.p12 } from Local Cluster node. Then generated each local node certificate { elastic-certificates.p12 } by using command :--

elasticsearch-certutil cert --ca elastic-stack-ca.p12 --ip <localnodes> , <remotenodes> -v

I copied , CA certificate and Node Certificate on each node. Then my Cluster was working fine as expected for cross-cluster search purpose.

So as per documentation :

a) Using the same certificate authority to generate certificates for all connected clusters, or
b) Adding the CA certificate from the local cluster as a trusted CA in each remote cluster.

I used Method a).

As for method b) -- I as not getting any solution to be how it should be implemented.

I will wait in her for few days , if i receive any response on b) method of implementation.

As for me production implementation has some time-window left yet.

Thanks
Tushar Nemade

TimV · December 30, 2020, 5:56am

You can copy trusted CAs between truststores using Java's keytool

You will need to run this twice, so that you can copy each CA from one cluster to the other.

keytool -importkeystore \
   -srckeystore /path/to/other-cluster/elastic-certificates.p12 -srcstorepass "" -srcalias "ca" \
   -destkeystore /path/to/this-cluster/elastic-certificates.p12 -deststorepass "" -destalias "remo
te-ca"

tusharnemade · December 30, 2020, 6:34am

Hi Tim

Thanks for your response.

I will execute now this and confirm in here if it worked or not.

tusharnemade · December 30, 2020, 12:36pm

Hi Tim

I did the execution of keytool as you mentioned in your post.

keytool -importkeystore -srckeystore /mnt/elasticsearch-7.8.0/config/certs/elastic-certificates-elk-localnode-1.p12 -srcstorepass "elastic" -destkeystore /mnt/elasticsearch-7.8.0/config/certs/elastic-certificates.p12  -deststorepass "elastic"
Importing keystore /mnt/elasticsearch-7.8.0/config/certs/elastic-certificates-elk-localnode-1.p12 to /mnt/elasticsearch-7.8.0/config/certs/elastic-certificates.p12...
Existing entry alias instance exists, overwrite? [no]:  no
Enter new alias name    (RETURN to cancel import for this entry):  elk-localnode-instance
Entry for alias instance successfully imported.
Existing entry alias ca exists, overwrite? [no]:  no
Enter new alias name    (RETURN to cancel import for this entry):  elk-localnode-ca
Entry for alias ca successfully imported.
Import command completed:  2 entries successfully imported, 0 entries failed or cancelled

After importing , I checked the keystore contents , it is showing me both server details , as expected.

But still I received an error message of :--

[WARN ][o.e.x.c.s.t.n.SecurityNetty4Transport] [node-1] client did not trust this server's certificate, closing connection Netty4TcpChannel{localAddress=0.0.0.0/0.0.0.0:9300, remoteAddress=/172.22.11.73:38922}

I did only importing of CA certificate from Local Cluster Node to Remote Cluster Node. As this is my use case and required as per elasticsearch documentation.

Could you please help me in understanding ... will different "ALIAS" names in Local Cluster Node Certificate and Remote Cluster Node Certificate will cause an issue ?

OR

There is some other reason for this error ...

Thanks
Tushar Nemade

TimV · December 30, 2020, 10:22pm

You need to establish trust on both clusters. The documentation is misleading here, I will get it fixed up.

tusharnemade · December 31, 2020, 3:13am

Hi Tim

Thanks for your response.

I will perform this from both side and update in here my results.

tusharnemade · December 31, 2020, 7:13am

Hi Tim

I have executed keytool command in following ways to have both nodes of different cluster each , have their CA certificates trusted.

On Local Cluster node

[esearch@elk-localnode-1 ~]$ keytool -importkeystore -srckeystore /mnt/elasticsearch-7.8.0/config/certs/elastic-certificates-elk-remotenode-1.p12 -srcstorepass "elastic" -destkeystore /mnt/elasticsearch-7.8.0/config/certs/elastic-certificates.p12  -deststorepass "elastic"

Importing keystore /mnt/elasticsearch-7.8.0/config/certs/elastic-certificates-elk-remotenode-1.p12 to /mnt/elasticsearch-7.8.0/config/certs/elastic-certificates.p12...
Existing entry alias instance exists, overwrite? [no]:  no
Enter new alias name    (RETURN to cancel import for this entry):  elk-remotenode-1-remote
Entry for alias instance successfully imported.
Existing entry alias ca exists, overwrite? [no]:  no
Enter new alias name    (RETURN to cancel import for this entry):  elk-remotenode-1-ca
Entry for alias ca successfully imported.
Import command completed:  2 entries successfully imported, 0 entries failed or cancelled
[esearch@elk-localnode-1 ~]$

On Remote Cluster Node

[esearch@elk-remotenode-1 ~]$ keytool -importkeystore -srckeystore /mnt/elasticsearch-7.8.0/config/certs/elastic-certificates-elk-localnode-1.p12 -srcstorepass "elastic" -destkeystore /mnt/elasticsearch-7.8.0/config/certs/elastic-certificates.p12  -deststorepass "elastic"

Importing keystore /mnt/elasticsearch-7.8.0/config/certs/elastic-certificates-elk-localnode-1.p12 to /mnt/elasticsearch-7.8.0/config/certs/elastic-certificates.p12...
Existing entry alias instance exists, overwrite? [no]:  no
Enter new alias name    (RETURN to cancel import for this entry):  elk-localnode-instance
Entry for alias instance successfully imported.
Existing entry alias ca exists, overwrite? [no]:  no
Enter new alias name    (RETURN to cancel import for this entry):  elk-localnode-ca
Entry for alias ca successfully imported.
Import command completed:  2 entries successfully imported, 0 entries failed or cancelled

It did worked successfully as expected.

I will accept it as a solution in this forum.

Could you please help me in one more thing ...

I am generating Nodes Certificates using

elasticsearch-certutil ca

How to pass Alias names for Instance and CA while creating certificates.

Thanks
Tushar Nemade

tusharnemade · December 31, 2020, 8:44am

Could you please help me in one more thing ...

I am generating Nodes Certificates using

elasticsearch-certutil ca

How to pass Alias names for Instance and CA while creating certificates.

Thanks
Tushar Nemade

system · January 28, 2021, 8:44am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.