Cross Cluster Search on ECK - how to set ca cert

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

Hi,

So we are playing around with ECK on multiple clusters in different region. We want to establish cross cluster search from one central cluster.

How can we set the ca-cert for our clusters during creation?
We followed the instructions here to set the http certificate but how do we set the CA cert?

https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-custom-http-certificate.html

This is the error we get right now.
{
"error": {
"root_cause": [
{
"type": "transport_exception",
"reason": "handshake failed because connection reset"
}
],
"type": "connect_transport_exception",
"reason": "[100.102.0.8:9300] general node connection failure",
"caused_by": {
"type": "transport_exception",
"reason": "handshake failed because connection reset"
}
},
"status": 500
}

2

Did you ever get this resolved?

3

Hi,

We ended up creating new clusters with the same CA. Instructions here are pretty good. It works fine now.

4

So you're not using the operator any more?

5

No, we gave up on it. I like the idea but its not fully there yet.

6

Sorry for the late reply.

To configure additional CA certificates you can just use the xpack.security.transport.ssl.certificate_authorities key as described in the Elasticsearch documentation, there is nothing ECK specific here to keep in mind other than mounting the CA certificates into the pod.

An example Elasticsearch manifest for ECK could then look like this:

apiVersion: elasticsearch.k8s.elastic.co/v1beta1
kind: Elasticsearch
metadata:
  name: cluster-one
spec:
  version: 7.4.0
  nodeSets:
  - name: default
    count: 1
    config:
      xpack.security.transport.ssl.certificate_authorities:
      - /usr/share/elasticsearch/config/remote/ca.crt
    podTemplate:
      spec:
        containers:
        - name: elasticsearch
          volumeMounts:
          - name: remote-certs
            mountPath: /usr/share/elasticsearch/config/remote
        volumes:
        - name: remote-certs
          secret:
            secretName: cluster-two-es-transport-certs-public

This assumes that a secret called cluster-two-es-transport-certs-public exists containing the CA certs of the other cluster you want to connect to. You will also have to configure the CA of this cluster on the other side to establish mutual trust.

2 Likes
7

Thanks!
Put that into the documentation too and next time I have some spare time I will implement that. :slight_smile:

8

It is maybe worth pointing out (for others reading this) that setting up the CAs is in itself not sufficient to configure remote clusters across different regions/k8s clusters. You also would have to make sure that all nodes in the remote cluster are reachable from the other cluster eg. via some form of router (all depending on your setup so hard to make a specific recommendation)

9

We had to dig deep to find the proxy setting for cross cluster search to work with Kubernetes.

cluster.remote.euwest1.proxy: "k8s-node-ip:32500"
cluster.remote.euwest1.seeds: "elastic-master.logging:9300"
cluster.remote.euwest1.skip_unavailable: true

We still have the issue of having to specify one Kubernetes node as proxy. Haven't gotten any working TCP load balancing. Amazon is our cloud right now.

10

So I've got the CA cert now being used by both clusters with the
xpack.security.transport.ssl.certificate_authorities config value.

However I'm still getting org.elasticsearch.transport.TransportException: handshake failed because connection reset as an error when trying to connect to the remote cluster.

I've just taken the CA cert from the remote cluster's secret and put it in the certificate_authorities for the other cluster.

11

Over on the remote cluster when the error occurs I see this error in the ES logs:

PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
12

How is the config for the two clusters? Somehow you messed up the certificates or the proxy thingy. Running in Kubernetes or on vms?

13

Ok, I got it figured out.

It was my mistake in mis-reading the docs. I thought it was only the querying cluster that needed the CA cert of the remote cluster added, but you need the CA of each added to the other.

@iremmats, thank you for the proxy hint, that bit me as well!

1 Like
14

Is there a way to "prime" the cluster with a pre-generated CA cert for transport, so that when the cluster comes up it uses a well-known CA to generate the cert secrets so I don't need to look it up after the cluster starts?

15

@glennslaven, for transport protocol we don't support user configured certificates. At least not at the moment with 1.0.

But even for HTTP, there's no way to put just only the CA (and key) and make the operator to sign the certificates with that CA. If custom certificates are used (for http) we expect the users to give all three: CA, cert and private key.

16

@pebrc can you elaborate on the router you are talking about?

I am attempting cross cluster replication where my leader exposes a k8s LB service on port 9300. However, I still cannot connect.

unexpected remote node {leader-es-default-1}

Scaling down Leader to 1 node will allow follower to connect to leader, but then I get the following error when I try to replicate an index.

No route to host: <leader cluster IP>/<leader cluster ip>:9300",
17

Is there any plans to have this by GA. We are looking at this for OpenShift and ECK without the ability to pass these to secrets to allow CCS and CCR seems like a deal breaker for many that don't use AWS, GCE, etc.