Elastic can't connect remote cluster

Verison

elasticsearch: 7.13.1

cluster-one and cluster-two have same config.

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: elasticsearch
  labels:
    app: elasticsearch
spec:
  selector:
    matchLabels:
      app: elasticsearch
  serviceName: "elasticsearch"
  replicas: 1
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: elasticsearch
        image: elasticsearch:7.13.1
        ports:
        - containerPort: 9200
          name: es-cli
        - containerPort: 9300
          name: es-iner
        env:
        - name: discovery.type
          value: single-node

The Use the cluster update settings API to dynamically configure remote settings on every node in the cluster-two.The remote seeds is cluster-one:9300

PUT _cluster/settings
{
  "persistent": {
    "cluster": {
      "remote": {
        "eks1": {
          "seeds": [
            "cluster-one:9300"
          ]
        }
      }
    }
  }
}

But GET _remote/info,is show

{
  "eks1" : {
    "connected" : false,
    "mode" : "sniff",
    "seeds" : [
      "cluster-one:9300"
    ],
    "num_nodes_connected" : 0,
    "max_connections_per_cluster" : 3,
    "initial_connect_timeout" : "30s",
    "skip_unavailable" : false
  }
}

I try to connect cluster-one:9300 on cluster-two node. Return This is not an HTTP
The cluster-one logs :

{"type": "server", "timestamp": "2021-09-01T07:20:21,021Z", "level": "INFO", "component": "o.e.c.s.ClusterSettings", "cluster.name": "cluster-two", "node.name": "elasticsearch-0", "message": "updating [cluster.remote.eks1.seeds] from [[]] to [[\"cluster-one:9300\"]]", "cluster.uuid": "NHNbWruqQmGJ7Y7yl8Kg7A", "node.id": "TNLCNX_sRFCRbIHJIUXutg"  }
{"type": "server", "timestamp": "2021-09-01T07:20:31,038Z", "level": "WARN", "component": "o.e.t.RemoteClusterService", "cluster.name": "cluster-two", "node.name": "elasticsearch-0", "message": "failed to connect to new remote cluster eks1 within 10s", "cluster.uuid": "NHNbWruqQmGJ7Y7yl8Kg7A", "node.id": "TNLCNX_sRFCRbIHJIUXutg"  }
{"type": "server", "timestamp": "2021-09-01T07:20:51,108Z", "level": "WARN", "component": "o.e.t.SniffConnectionStrategy", "cluster.name": "cluster-two", "node.name": "elasticsearch-0", "message": "fetching nodes from external cluster [eks1] failed", "cluster.uuid": "NHNbWruqQmGJ7Y7yl8Kg7A", "node.id": "TNLCNX_sRFCRbIHJIUXutg" ,
"stacktrace": ["org.elasticsearch.transport.ConnectTransportException: [elasticsearch-0][xxxxxx:9300] connect_timeout[30s]",
"at org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onTimeout(TcpTransport.java:977) ~[elasticsearch-7.13.1.jar:7.13.1]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:673) ~[elasticsearch-7.13.1.jar:7.13.1]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]",
"at java.lang.Thread.run(Thread.java:831) [?:?]"] }

I don't know what is the wrong.How to fix it?

Just so I am following here, are you setting up CCS or CCR here?

PUT _cluster/settings
{
  "persistent": {
    "cluster": {
      "remote": {
        "eks1": {
          "seeds": [
            "cluster-one:9300"
          ]
        }
      }
    }
  }
}

s there a problem with my understanding :rofl:? This step is setting up CCS

kibana setting remote cluster

The remote cluster is single-node, I setup the transport.tcp.port=9300

  containers:
      - name: elasticsearch
        image: elasticsearch:7.13.1
        ports:
        - containerPort: 9200
          name: es-cli
        - containerPort: 9300
          name: es-iner
        env:
        - name: discovery.type
          value: single-node
        - name: cluster.name
          value: eks1
        - name: transport.tcp.port
          value: '9300'

I fix it. Because Elastic is created in a different K8S cluster, it cannot be resolved to a specific ES address through SVC, so it is always disconnected