Cross Cluster Replication on ECK - cross region replication

Hey,

Big fan of the ECK v1 beta! but I could use some help on how to do cross-cluster replication with two AKS clusters in different regions.

  1. When using the leader's IP address as cluster.remote.leader.proxy and cluster.remote.leader.seeds on the follower cluster, I am unable to connect to the remote cluster. This will fail with the following message - "unexpected remote node {-es-default-1}". So I scaled down my leader to 1 node and my follower was able to connect! Is 1 node a limitation of CCR? Any chance someone could shed some light on this?

  2. Unfortunately, even though my follower is now connected to my leader, replication still does not work. When I try to create a follower index, my follower errors out saying something along the lines of "No route to host: /:9300".

If someone has successfully done CCR with a remote k8s cluster, I would love a sample.

This is what I have so far on the follower:

    apiVersion: elasticsearch.k8s.elastic.co/v1beta1
    kind: Elasticsearch
    metadata:
      name: search-follower
    spec:
      version: 7.4.0
      http:
        service:
          spec:
            type: LoadBalancer
        tls:
          selfSignedCertificate:
            disabled: true
      nodeSets:
      - name: default
        count: 2
        config:
          node.master: true
          node.data: true
          node.ingest: true
          node.store.allow_mmap: false
          node.attr.server_name: ${POD_NAME}.node.search-follower.default.es.cluster.local
          cluster.remote.search-leader.proxy: "<leader LB ip>:9300"
          cluster.remote.search-leader.seeds: "<leader LB ip>:9300"
          xpack.security.transport.ssl.certificate_authorities:
          - /usr/share/elasticsearch/config/other/ca.crt
        podTemplate:
          spec:
            containers:
            - name: elasticsearch
              volumeMounts:
              - name: search-leader-certs
                mountPath: /usr/share/elasticsearch/config/other
            volumes:
            - name: search-leader-certs
              secret:
                secretName: search-leader-es-transport-certs-public
        volumeClaimTemplates:
        - metadata:
            name: elasticsearch-data
          spec:
            accessModes:
            - ReadWriteOnce
            resources:
              requests:
                storage: 5Gi
            storageClassName: default