# Remote cluster connected but not listed

**URL:** <https://discuss.elastic.co/t/remote-cluster-connected-but-not-listed/238203>\
**Category:** Elasticsearch\
**Created:** [June 23, 2020, 7:13am UTC](https://discuss.elastic.co/t/remote-cluster-connected-but-not-listed/238203 "2020-06-23T07:13:31Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Sebastian\_Treu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sebastian_treu/32/27013_2.png) [@Sebastian\_Treu](https://discuss.elastic.co/u/Sebastian_Treu)\
**Post date:** [June 23, 2020, 7:13am UTC](https://discuss.elastic.co/t/remote-cluster-connected-but-not-listed/238203/1 "2020-06-23T07:13:31Z")

</div>

Hey there, I'm trying to connect a local cluster (7.8.0) with a remote cluster (7.3.2) where both has secured `xpack.security.transport.ssl.enabled=true` with the same certificate, key and ca, and both are deployed using docker (swarm mode) in different cloud providers.

### Certificate, key and CA

They are the same for both clusters, however in local cluster they are in .pem format extracted from pcks12 with:

```auto
openssl pkcs12 -in elasticsearch-certificates.p12 -cacerts -nokeys -out ca.crt
openssl pkcs12 -in elasticsearch-certificates.p12 -clcerts -nokeys -out node.crt
openssl pkcs12 -in elasticsearch-certificates.p12 -nocerts -out node.pem

```

### Remote cluster (7.3.2)

Remote cluster is an ELK stack with 3 masters nodes and 1 coordinator node which is used for kibana to connect to. Kibana is proxied by traefik.

> **remote cluster master node config**
>
> ```
> - cluster.name=remote.cluster
> - node.name=es1
> - "ES_JAVA_OPTS=-Xms6g -Xmx6g"
> - network.host=_eth0:ipv4_,_eth1:ipv4_,_local_
> - transport.host=_eth0:ipv4_,_eth1:ipv4_,_local_
> - transport.bind_host=_eth0:ipv4_,_eth1:ipv4_,_local_
> - transport.port=9300
> - cluster.remote.connect=true
> - path.repo=/usr/share/elasticsearch/backup
> - gateway.expected_nodes=3
> - discovery.zen.minimum_master_nodes=2
> - discovery.zen.ping.unicast.hosts=elasticsearch-node3,elasticsearch-node2
> - xpack.monitoring.collection.interval=30s
> - xpack.security.transport.ssl.enabled=true
> - xpack.security.transport.ssl.verification_mode=certificate
> - xpack.security.transport.ssl.keystore.path=/usr/share/elasticsearch/config/certs/elastic-certificates.p12
> - xpack.security.transport.ssl.truststore.path=/usr/share/elasticsearch/config/certs/elastic-certificates.p12
> 
> ```

### Local cluster (7.8.0)

Local cluster is actually the same with 7.8.0 minor customizations removing deprecated configs.

> **localcluster master node config**
>
> ```
> - cluster.name=cluster
> - cluster.initial_master_nodes=es1,es2,es3,es-coordinator
> - discovery.seed_hosts=elasticsearch-node1,elasticsearch-node2,elasticsearch-node3
> - node.name=es1
> - "ES_JAVA_OPTS=-Xms6g -Xmx6g"
> - network.host=_eth0:ipv4_,_eth1:ipv4_,_local_
> - transport.host=_eth0:ipv4_,_eth1:ipv4_,_local_
> - transport.bind_host=_eth0:ipv4_,_eth1:ipv4_,_local_
> - transport.tcp.port=9300
> - indices.recovery.max_bytes_per_sec=20mb
> - path.repo=/usr/share/elasticsearch/backup
> - gateway.expected_data_nodes=3
> - cluster.routing.allocation.node_concurrent_recoveries=1
> - xpack.monitoring.collection.interval=30s
> - xpack.security.transport.ssl.enabled=true
> - xpack.security.transport.ssl.verification_mode=certificate
> - xpack.security.transport.ssl.key=/usr/share/elasticsearch/config/certs/key.pem
> - xpack.security.transport.ssl.certificate=/usr/share/elasticsearch/config/certs/crt.pem
> - xpack.security.transport.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/ca.crt
> 
> ```

### Remote cluster datacenter

I have exposed port 9300 in the remote cluster to 1 elasticsearch master node. So, `PUBLIC_IP:9300` connects directly into the master node. Using the certificate and key, I have checked:

`openssl s_client -key key.pem -cert cet.pem PUBLIC_IP:9300`

and certificate and key are valid.

### Kibana add remote cluster (or PUT \_cluster/settings)

In the local cluster I have configured remote cluster with proxy mode and proxy address `PUBLIC_IP:9300` and 1 proxy connection (for easy debugging/tracing)

So far so good. But nothing happens. No info on `GET _remote/info` and I see no remote cluster in kibana dashboard. I was able to log `TRACE` connections greping the IP and I was able to catch `DEBUG` log that shows that local cluster was connected to remote cluster:

```auto
{"type": "server", "timestamp": "2020-06-23T05:43:45,168Z", "level": "DEBUG", "component": "o.e.t.ClusterConnectionManager", "cluster.name": "cluster", "node.name": "es3", "message": "connected to node [{remote#PUBLIC_IP:9300}{Xjq02bN9Tye7GG-N5Ysbxw}{PUBLIC_IP}{PUBLIC_IP:9300}{dimr}]", "cluster.uuid": "cluster-uuid", "node.id": "node-id" }

```

Followin is the `TRACE` log of the transport. I have removed the requests body because it looks like text can be obtained from it.

> **transport TRACE logs**
>
> ```
> {"type": "server", "timestamp": "2020-06-23T06:52:15,531Z", "level": "INFO", "component": "o.e.c.s.ClusterSettings", "cluster.name": "cluster", "node.name": "es2", "message": "updating [cluster.remote.rem.proxy_address] from [] to [PUBLIC_IP:9300]", "cluster.uuid": "cluster-uuid", "node.id": "node-id" }
> {"type": "server", "timestamp": "2020-06-23T06:52:15,534Z", "level": "INFO", "component": "o.e.c.s.ClusterSettings", "cluster.name": "cluster", "node.name": "es2", "message": "updating [cluster.remote.rem.proxy_address] from [] to [PUBLIC_IP:9300]", "cluster.uuid": "cluster-uuid", "node.id": "node-id" }
> {"type": "server", "timestamp": "2020-06-23T06:52:15,536Z", "level": "INFO", "component": "o.e.c.s.ClusterSettings", "cluster.name": "cluster", "node.name": "es2", "message": "updating [cluster.remote.rem.proxy_address] from [] to [PUBLIC_IP:9300]", "cluster.uuid": "cluster-uuid", "node.id": "node-id" }
> {"type": "server", "timestamp": "2020-06-23T06:52:15,538Z", "level": "INFO", "component": "o.e.c.s.ClusterSettings", "cluster.name": "cluster", "node.name": "es2", "message": "updating [cluster.remote.rem.proxy_address] from [] to [PUBLIC_IP:9300]", "cluster.uuid": "cluster-uuid", "node.id": "node-id" }
> {"type": "server", "timestamp": "2020-06-23T06:52:16,608Z", "level": "INFO", "component": "o.e.c.s.ClusterSettings", "cluster.name": "cluster", "node.name": "es2", "message": "updating [cluster.remote.rem.proxy_address] from [] to [PUBLIC_IP:9300]", "cluster.uuid": "cluster-uuid", "node.id": "node-id" }
> {"type": "server", "timestamp": "2020-06-23T06:52:16,612Z", "level": "INFO", "component": "o.e.c.s.ClusterSettings", "cluster.name": "cluster", "node.name": "es2", "message": "updating [cluster.remote.rem.proxy_address] from [] to [PUBLIC_IP:9300]", "cluster.uuid": "cluster-uuid", "node.id": "node-id" }
> {"type": "server", "timestamp": "2020-06-23T06:52:16,614Z", "level": "INFO", "component": "o.e.c.s.ClusterSettings", "cluster.name": "cluster", "node.name": "es2", "message": "updating [cluster.remote.rem.proxy_address] from [] to [PUBLIC_IP:9300]", "cluster.uuid": "cluster-uuid", "node.id": "node-id" }
> {"type": "server", "timestamp": "2020-06-23T06:52:16,616Z", "level": "INFO", "component": "o.e.c.s.ClusterSettings", "cluster.name": "cluster", "node.name": "es2", "message": "updating [cluster.remote.rem.proxy_address] from [] to [PUBLIC_IP:9300]", "cluster.uuid": "cluster-uuid", "node.id": "node-id" }
> {"type": "server", "timestamp": "2020-06-23T06:52:16,624Z", "level": "TRACE", "component": "o.e.t.n.ESLoggingHandler", "cluster.name": "cluster", "node.name": "es2", "message": "[id: 0x77cf362d] CONNECT: /PUBLIC_IP:9300", "cluster.uuid": "cluster-uuid", "node.id": "node-id" }
> {"type": "server", "timestamp": "2020-06-23T06:52:16,798Z", "level": "TRACE", "component": "o.e.t.TransportLogger", "cluster.name": "cluster", "node.name": "es2", "message": "Netty4TcpChannel{localAddress=/172.18.0.3:34464, remoteAddress=/PUBLIC_IP:9300} [length: 58, request id: 31638, type: request, version: 6.8.0, action: internal:tcp/handshake] WRITE: 58B", "cluster.uuid": "cluster-uuid", "node.id": "node-id" }
> {"type": "server", "timestamp": "2020-06-23T06:52:16,799Z", "level": "TRACE", "component": "o.e.t.n.ESLoggingHandler", "cluster.name": "cluster", "node.name": "es2", "message": "[id: 0x77cf362d, L:/172.18.0.3:34464 - R:/PUBLIC_IP:9300] WRITE: 58B\n +-------------------------------------------------+\n | BODY REMOVED |\n+--------+-------------------------------------------------+----------------+", "cluster.uuid": "cluster-uuid", "node.id": "node-id" }
> {"type": "server", "timestamp": "2020-06-23T06:52:16,800Z", "level": "TRACE", "component": "o.e.t.n.ESLoggingHandler", "cluster.name": "cluster", "node.name": "es2", "message": "[id: 0x77cf362d, L:/172.18.0.3:34464 - R:/PUBLIC_IP:9300] FLUSH", "cluster.uuid": "cluster-uuid", "node.id": "node-id" }
> {"type": "server", "timestamp": "2020-06-23T06:52:16,817Z", "level": "TRACE", "component": "o.e.t.n.ESLoggingHandler", "cluster.name": "cluster", "node.name": "es2", "message": "[id: 0x77cf362d, L:/172.18.0.3:34464 - R:/PUBLIC_IP:9300] ACTIVE", "cluster.uuid": "cluster-uuid", "node.id": "node-id" }
> {"type": "server", "timestamp": "2020-06-23T06:52:17,218Z", "level": "TRACE", "component": "o.e.t.n.ESLoggingHandler", "cluster.name": "cluster", "node.name": "es2", "message": "[id: 0x77cf362d, L:/172.18.0.3:34464 - R:/PUBLIC_IP:9300] USER_EVENT: SslHandshakeCompletionEvent(SUCCESS)", "cluster.uuid": "cluster-uuid", "node.id": "node-id" }
> {"type": "server", "timestamp": "2020-06-23T06:52:17,395Z", "level": "TRACE", "component": "o.e.t.n.ESLoggingHandler", "cluster.name": "cluster", "node.name": "es2", "message": "[id: 0x77cf362d, L:/172.18.0.3:34464 - R:/PUBLIC_IP:9300] USER_EVENT: SslHandshakeCompletionEvent(SUCCESS)", "cluster.uuid": "cluster-uuid", "node.id": "node-id" }
> {"type": "server", "timestamp": "2020-06-23T06:52:17,396Z", "level": "TRACE", "component": "o.e.t.n.ESLoggingHandler", "cluster.name": "cluster", "node.name": "es2", "message": "[id: 0x77cf362d, L:/172.18.0.3:34464 - R:/PUBLIC_IP:9300] READ: 25B\n +-------------------------------------------------+\n | BODY REMOVED |\n+--------+-------------------------------------------------+----------------+", "cluster.uuid": "cluster-uuid", "node.id": "node-id" }
> {"type": "server", "timestamp": "2020-06-23T06:52:17,396Z", "level": "TRACE", "component": "o.e.t.TransportLogger", "cluster.name": "cluster", "node.name": "es2", "message": "Netty4TcpChannel{localAddress=/172.18.0.3:34464, remoteAddress=/PUBLIC_IP:9300} [length: 25, request id: 31638, type: response, version: 6.8.0] READ: 25B", "cluster.uuid": "cluster-uuid", "node.id": "node-id" }
> {"type": "server", "timestamp": "2020-06-23T06:52:17,397Z", "level": "TRACE", "component": "o.e.t.TransportLogger", "cluster.name": "cluster", "node.name": "es2", "message": "Netty4TcpChannel{localAddress=/172.18.0.3:34464, remoteAddress=/PUBLIC_IP:9300} [length: 143, request id: 31642, type: request, version: 7.3.2, action: internal:transport/handshake] WRITE: 143B", "cluster.uuid": "cluster-uuid", "node.id": "node-id" }
> {"type": "server", "timestamp": "2020-06-23T06:52:17,397Z", "level": "TRACE", "component": "o.e.t.n.ESLoggingHandler", "cluster.name": "cluster", "node.name": "es2", "message": "[id: 0x77cf362d, L:/172.18.0.3:34464 - R:/PUBLIC_IP:9300] WRITE: 143B\n +-------------------------------------------------+\n | BODY REMOVED |port/handshake. |\n+--------+-------------------------------------------------+----------------+", "cluster.uuid": "cluster-uuid", "node.id": "node-id" }
> {"type": "server", "timestamp": "2020-06-23T06:52:17,397Z", "level": "TRACE", "component": "o.e.t.n.ESLoggingHandler", "cluster.name": "cluster", "node.name": "es2", "message": "[id: 0x77cf362d, L:/172.18.0.3:34464 - R:/PUBLIC_IP:9300] FLUSH", "cluster.uuid": "cluster-uuid", "node.id": "node-id" }
> {"type": "server", "timestamp": "2020-06-23T06:52:17,570Z", "level": "TRACE", "component": "o.e.t.n.ESLoggingHandler", "cluster.name": "cluster", "node.name": "es2", "message": "[id: 0x77cf362d, L:/172.18.0.3:34464 - R:/PUBLIC_IP:9300] READ: 360B\n +-------------------------------------------------+\n | BODY REMOVD |\n+--------+-------------------------------------------------+----------------+", "cluster.uuid": "cluster-uuid", "node.id": "node-id" }
> {"type": "server", "timestamp": "2020-06-23T06:52:17,571Z", "level": "TRACE", "component": "o.e.t.TransportLogger", "cluster.name": "cluster", "node.name": "es2", "message": "Netty4TcpChannel{localAddress=/172.18.0.3:34464, remoteAddress=/PUBLIC_IP:9300} [length: 360, request id: 31642, type: response, version: 7.3.2] READ: 360B", "cluster.uuid": "cluster-uuid", "node.id": "node-id" }
> {"type": "server", "timestamp": "2020-06-23T06:52:17,571Z", "level": "DEBUG", "component": "o.e.t.ClusterConnectionManager", "cluster.name": "cluster", "node.name": "es2", "message": "connected to node [{rem#PUBLIC_IP:9300}{95NcT6yjRLOvZTSBtL4ozg}{PUBLIC_IP}{PUBLIC_IP:9300}{dimr}]", "cluster.uuid": "cluster-uuid", "node.id": "node-id" }
> 
> ```

Honestly, I don't know what am I missing. This is day 3 of reading documentation and looks like it should be a simple procedure. I'm not sure how proxy mode works, but from logs looks like a handshake was made and a connection was succesful.

Any help please?

---

<div class="post-metadata">

**Author:** ![Sebastian\_Treu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sebastian_treu/32/27013_2.png) [@Sebastian\_Treu](https://discuss.elastic.co/u/Sebastian_Treu)\
**Post date:** [July 10, 2020, 3:14am UTC](https://discuss.elastic.co/t/remote-cluster-connected-but-not-listed/238203/2 "2020-07-10T03:14:47Z")

</div>

Anyone??

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 7, 2020, 3:14am UTC](https://discuss.elastic.co/t/remote-cluster-connected-but-not-listed/238203/3 "2020-08-07T03:14:48Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.