Remote cluster connected but not listed

Elastic Stack Elasticsearch
1

Hey there, I'm trying to connect a local cluster (7.8.0) with a remote cluster (7.3.2) where both has secured xpack.security.transport.ssl.enabled=true with the same certificate, key and ca, and both are deployed using docker (swarm mode) in different cloud providers.

Certificate, key and CA

They are the same for both clusters, however in local cluster they are in .pem format extracted from pcks12 with:

openssl pkcs12 -in elasticsearch-certificates.p12 -cacerts -nokeys -out ca.crt
openssl pkcs12 -in elasticsearch-certificates.p12 -clcerts -nokeys -out node.crt
openssl pkcs12 -in elasticsearch-certificates.p12 -nocerts -out node.pem

Remote cluster (7.3.2)

Remote cluster is an ELK stack with 3 masters nodes and 1 coordinator node which is used for kibana to connect to. Kibana is proxied by traefik.

remote cluster master node config 
  - cluster.name=remote.cluster
  - node.name=es1
  - "ES_JAVA_OPTS=-Xms6g -Xmx6g"
  - network.host=_eth0:ipv4_,_eth1:ipv4_,_local_
  - transport.host=_eth0:ipv4_,_eth1:ipv4_,_local_
  - transport.bind_host=_eth0:ipv4_,_eth1:ipv4_,_local_
  - transport.port=9300
  - cluster.remote.connect=true
  - path.repo=/usr/share/elasticsearch/backup
  - gateway.expected_nodes=3
  - discovery.zen.minimum_master_nodes=2
  - discovery.zen.ping.unicast.hosts=elasticsearch-node3,elasticsearch-node2
  - xpack.monitoring.collection.interval=30s
  - xpack.security.transport.ssl.enabled=true
  - xpack.security.transport.ssl.verification_mode=certificate
  - xpack.security.transport.ssl.keystore.path=/usr/share/elasticsearch/config/certs/elastic-certificates.p12
  - xpack.security.transport.ssl.truststore.path=/usr/share/elasticsearch/config/certs/elastic-certificates.p12

Local cluster (7.8.0)

Local cluster is actually the same with 7.8.0 minor customizations removing deprecated configs.

localcluster master node config 
- cluster.name=cluster
- cluster.initial_master_nodes=es1,es2,es3,es-coordinator
- discovery.seed_hosts=elasticsearch-node1,elasticsearch-node2,elasticsearch-node3
- node.name=es1
- "ES_JAVA_OPTS=-Xms6g -Xmx6g"
- network.host=_eth0:ipv4_,_eth1:ipv4_,_local_
- transport.host=_eth0:ipv4_,_eth1:ipv4_,_local_
- transport.bind_host=_eth0:ipv4_,_eth1:ipv4_,_local_
- transport.tcp.port=9300
- indices.recovery.max_bytes_per_sec=20mb
- path.repo=/usr/share/elasticsearch/backup
- gateway.expected_data_nodes=3
- cluster.routing.allocation.node_concurrent_recoveries=1
- xpack.monitoring.collection.interval=30s
- xpack.security.transport.ssl.enabled=true
- xpack.security.transport.ssl.verification_mode=certificate
- xpack.security.transport.ssl.key=/usr/share/elasticsearch/config/certs/key.pem
- xpack.security.transport.ssl.certificate=/usr/share/elasticsearch/config/certs/crt.pem
- xpack.security.transport.ssl.certificate_authorities=/usr/share/elasticsearch/config/certs/ca.crt

Remote cluster datacenter

I have exposed port 9300 in the remote cluster to 1 elasticsearch master node. So, PUBLIC_IP:9300 connects directly into the master node. Using the certificate and key, I have checked:

openssl s_client -key key.pem -cert cet.pem PUBLIC_IP:9300

and certificate and key are valid.

Kibana add remote cluster (or PUT _cluster/settings)

In the local cluster I have configured remote cluster with proxy mode and proxy address PUBLIC_IP:9300 and 1 proxy connection (for easy debugging/tracing)

So far so good. But nothing happens. No info on GET _remote/info and I see no remote cluster in kibana dashboard. I was able to log TRACE connections greping the IP and I was able to catch DEBUG log that shows that local cluster was connected to remote cluster:

{"type": "server", "timestamp": "2020-06-23T05:43:45,168Z", "level": "DEBUG", "component": "o.e.t.ClusterConnectionManager", "cluster.name": "cluster", "node.name": "es3", "message": "connected to node [{remote#PUBLIC_IP:9300}{Xjq02bN9Tye7GG-N5Ysbxw}{PUBLIC_IP}{PUBLIC_IP:9300}{dimr}]", "cluster.uuid": "cluster-uuid", "node.id": "node-id"  }

Followin is the TRACE log of the transport. I have removed the requests body because it looks like text can be obtained from it.

transport TRACE logs 
{"type": "server", "timestamp": "2020-06-23T06:52:15,531Z", "level": "INFO", "component": "o.e.c.s.ClusterSettings", "cluster.name": "cluster", "node.name": "es2", "message": "updating [cluster.remote.rem.proxy_address] from [] to [PUBLIC_IP:9300]", "cluster.uuid": "cluster-uuid", "node.id": "node-id"  }
{"type": "server", "timestamp": "2020-06-23T06:52:15,534Z", "level": "INFO", "component": "o.e.c.s.ClusterSettings", "cluster.name": "cluster", "node.name": "es2", "message": "updating [cluster.remote.rem.proxy_address] from [] to [PUBLIC_IP:9300]", "cluster.uuid": "cluster-uuid", "node.id": "node-id"  }
{"type": "server", "timestamp": "2020-06-23T06:52:15,536Z", "level": "INFO", "component": "o.e.c.s.ClusterSettings", "cluster.name": "cluster", "node.name": "es2", "message": "updating [cluster.remote.rem.proxy_address] from [] to [PUBLIC_IP:9300]", "cluster.uuid": "cluster-uuid", "node.id": "node-id"  }
{"type": "server", "timestamp": "2020-06-23T06:52:15,538Z", "level": "INFO", "component": "o.e.c.s.ClusterSettings", "cluster.name": "cluster", "node.name": "es2", "message": "updating [cluster.remote.rem.proxy_address] from [] to [PUBLIC_IP:9300]", "cluster.uuid": "cluster-uuid", "node.id": "node-id"  }
{"type": "server", "timestamp": "2020-06-23T06:52:16,608Z", "level": "INFO", "component": "o.e.c.s.ClusterSettings", "cluster.name": "cluster", "node.name": "es2", "message": "updating [cluster.remote.rem.proxy_address] from [] to [PUBLIC_IP:9300]", "cluster.uuid": "cluster-uuid", "node.id": "node-id"  }
{"type": "server", "timestamp": "2020-06-23T06:52:16,612Z", "level": "INFO", "component": "o.e.c.s.ClusterSettings", "cluster.name": "cluster", "node.name": "es2", "message": "updating [cluster.remote.rem.proxy_address] from [] to [PUBLIC_IP:9300]", "cluster.uuid": "cluster-uuid", "node.id": "node-id"  }
{"type": "server", "timestamp": "2020-06-23T06:52:16,614Z", "level": "INFO", "component": "o.e.c.s.ClusterSettings", "cluster.name": "cluster", "node.name": "es2", "message": "updating [cluster.remote.rem.proxy_address] from [] to [PUBLIC_IP:9300]", "cluster.uuid": "cluster-uuid", "node.id": "node-id"  }
{"type": "server", "timestamp": "2020-06-23T06:52:16,616Z", "level": "INFO", "component": "o.e.c.s.ClusterSettings", "cluster.name": "cluster", "node.name": "es2", "message": "updating [cluster.remote.rem.proxy_address] from [] to [PUBLIC_IP:9300]", "cluster.uuid": "cluster-uuid", "node.id": "node-id"  }
{"type": "server", "timestamp": "2020-06-23T06:52:16,624Z", "level": "TRACE", "component": "o.e.t.n.ESLoggingHandler", "cluster.name": "cluster", "node.name": "es2", "message": "[id: 0x77cf362d] CONNECT: /PUBLIC_IP:9300", "cluster.uuid": "cluster-uuid", "node.id": "node-id"  }
{"type": "server", "timestamp": "2020-06-23T06:52:16,798Z", "level": "TRACE", "component": "o.e.t.TransportLogger", "cluster.name": "cluster", "node.name": "es2", "message": "Netty4TcpChannel{localAddress=/172.18.0.3:34464, remoteAddress=/PUBLIC_IP:9300} [length: 58, request id: 31638, type: request, version: 6.8.0, action: internal:tcp/handshake] WRITE: 58B", "cluster.uuid": "cluster-uuid", "node.id": "node-id"  }
{"type": "server", "timestamp": "2020-06-23T06:52:16,799Z", "level": "TRACE", "component": "o.e.t.n.ESLoggingHandler", "cluster.name": "cluster", "node.name": "es2", "message": "[id: 0x77cf362d, L:/172.18.0.3:34464 - R:/PUBLIC_IP:9300] WRITE: 58B\n         +-------------------------------------------------+\n         |  BODY REMOVED |\n+--------+-------------------------------------------------+----------------+", "cluster.uuid": "cluster-uuid", "node.id": "node-id"  }
{"type": "server", "timestamp": "2020-06-23T06:52:16,800Z", "level": "TRACE", "component": "o.e.t.n.ESLoggingHandler", "cluster.name": "cluster", "node.name": "es2", "message": "[id: 0x77cf362d, L:/172.18.0.3:34464 - R:/PUBLIC_IP:9300] FLUSH", "cluster.uuid": "cluster-uuid", "node.id": "node-id"  }
{"type": "server", "timestamp": "2020-06-23T06:52:16,817Z", "level": "TRACE", "component": "o.e.t.n.ESLoggingHandler", "cluster.name": "cluster", "node.name": "es2", "message": "[id: 0x77cf362d, L:/172.18.0.3:34464 - R:/PUBLIC_IP:9300] ACTIVE", "cluster.uuid": "cluster-uuid", "node.id": "node-id"  }
{"type": "server", "timestamp": "2020-06-23T06:52:17,218Z", "level": "TRACE", "component": "o.e.t.n.ESLoggingHandler", "cluster.name": "cluster", "node.name": "es2", "message": "[id: 0x77cf362d, L:/172.18.0.3:34464 - R:/PUBLIC_IP:9300] USER_EVENT: SslHandshakeCompletionEvent(SUCCESS)", "cluster.uuid": "cluster-uuid", "node.id": "node-id"  }
{"type": "server", "timestamp": "2020-06-23T06:52:17,395Z", "level": "TRACE", "component": "o.e.t.n.ESLoggingHandler", "cluster.name": "cluster", "node.name": "es2", "message": "[id: 0x77cf362d, L:/172.18.0.3:34464 - R:/PUBLIC_IP:9300] USER_EVENT: SslHandshakeCompletionEvent(SUCCESS)", "cluster.uuid": "cluster-uuid", "node.id": "node-id"  }
{"type": "server", "timestamp": "2020-06-23T06:52:17,396Z", "level": "TRACE", "component": "o.e.t.n.ESLoggingHandler", "cluster.name": "cluster", "node.name": "es2", "message": "[id: 0x77cf362d, L:/172.18.0.3:34464 - R:/PUBLIC_IP:9300] READ: 25B\n         +-------------------------------------------------+\n         |  BODY REMOVED |\n+--------+-------------------------------------------------+----------------+", "cluster.uuid": "cluster-uuid", "node.id": "node-id"  }
{"type": "server", "timestamp": "2020-06-23T06:52:17,396Z", "level": "TRACE", "component": "o.e.t.TransportLogger", "cluster.name": "cluster", "node.name": "es2", "message": "Netty4TcpChannel{localAddress=/172.18.0.3:34464, remoteAddress=/PUBLIC_IP:9300} [length: 25, request id: 31638, type: response, version: 6.8.0] READ: 25B", "cluster.uuid": "cluster-uuid", "node.id": "node-id"  }
{"type": "server", "timestamp": "2020-06-23T06:52:17,397Z", "level": "TRACE", "component": "o.e.t.TransportLogger", "cluster.name": "cluster", "node.name": "es2", "message": "Netty4TcpChannel{localAddress=/172.18.0.3:34464, remoteAddress=/PUBLIC_IP:9300} [length: 143, request id: 31642, type: request, version: 7.3.2, action: internal:transport/handshake] WRITE: 143B", "cluster.uuid": "cluster-uuid", "node.id": "node-id"  }
{"type": "server", "timestamp": "2020-06-23T06:52:17,397Z", "level": "TRACE", "component": "o.e.t.n.ESLoggingHandler", "cluster.name": "cluster", "node.name": "es2", "message": "[id: 0x77cf362d, L:/172.18.0.3:34464 - R:/PUBLIC_IP:9300] WRITE: 143B\n         +-------------------------------------------------+\n         |  BODY REMOVED  |port/handshake. |\n+--------+-------------------------------------------------+----------------+", "cluster.uuid": "cluster-uuid", "node.id": "node-id"  }
{"type": "server", "timestamp": "2020-06-23T06:52:17,397Z", "level": "TRACE", "component": "o.e.t.n.ESLoggingHandler", "cluster.name": "cluster", "node.name": "es2", "message": "[id: 0x77cf362d, L:/172.18.0.3:34464 - R:/PUBLIC_IP:9300] FLUSH", "cluster.uuid": "cluster-uuid", "node.id": "node-id"  }
{"type": "server", "timestamp": "2020-06-23T06:52:17,570Z", "level": "TRACE", "component": "o.e.t.n.ESLoggingHandler", "cluster.name": "cluster", "node.name": "es2", "message": "[id: 0x77cf362d, L:/172.18.0.3:34464 - R:/PUBLIC_IP:9300] READ: 360B\n         +-------------------------------------------------+\n         |  BODY REMOVD    |\n+--------+-------------------------------------------------+----------------+", "cluster.uuid": "cluster-uuid", "node.id": "node-id"  }
{"type": "server", "timestamp": "2020-06-23T06:52:17,571Z", "level": "TRACE", "component": "o.e.t.TransportLogger", "cluster.name": "cluster", "node.name": "es2", "message": "Netty4TcpChannel{localAddress=/172.18.0.3:34464, remoteAddress=/PUBLIC_IP:9300} [length: 360, request id: 31642, type: response, version: 7.3.2] READ: 360B", "cluster.uuid": "cluster-uuid", "node.id": "node-id"  }
{"type": "server", "timestamp": "2020-06-23T06:52:17,571Z", "level": "DEBUG", "component": "o.e.t.ClusterConnectionManager", "cluster.name": "cluster", "node.name": "es2", "message": "connected to node [{rem#PUBLIC_IP:9300}{95NcT6yjRLOvZTSBtL4ozg}{PUBLIC_IP}{PUBLIC_IP:9300}{dimr}]", "cluster.uuid": "cluster-uuid", "node.id": "node-id"  }

Honestly, I don't know what am I missing. This is day 3 of reading documentation and looks like it should be a simple procedure. I'm not sure how proxy mode works, but from logs looks like a handshake was made and a connection was succesful.

Any help please?

2

Anyone??

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.