I get an error when connecting 2 remote clusters

Elastic Stack Elasticsearch
1

However, I am encountering an issue when trying to connect Cluster 1 to Cluster 2 as a remote cluster. The nodes on both clusters are trying to communicate using their internal IP addresses (the IPs assigned to each container), but the internal IPs of the containers on the two clusters are different and cannot be on the same network, which is causing connectivity issues.

I added network.host=0.0.0.0 and transport.host=0.0.0.0 to allow connections from any network interface, but the nodes still seem to attempt to use the local IP addresses. I also observe routing issues between the nodes on the two VMs. Is there a way for nodes on different VMs to communicate with each other? I need guidance on how to resolve this.

Elasticsearch logs on Cluster 1 show the following:

{"@timestamp":"2025-02-25T10:18:47.111Z", "log.level": "INFO", "message":"updating [cluster.remote.RC.mode] from [SNIFF] to [sniff]", "elasticsearch.cluster.name":"cluster1"}
{"@timestamp":"2025-02-25T10:18:47.112Z", "log.level": "INFO", "message":"updating [cluster.remote.RC.seeds] from to [["192.168.70.129:9300"]]", "elasticsearch.cluster.name":"cluster1"}
{"@timestamp":"2025-02-25T10:18:50.543Z", "log.level": "WARN", "message":"fetching nodes from external cluster [RC] failed", "error.message":"[elasticsearch][10.1.1.6:9300] connect_exception"}

Elasticsearch logs on Cluster 2 show the following:

{"@timestamp":"2025-02-25T10:19:27.098Z", "log.level": "WARN", "message":"received plaintext traffic on an encrypted channel, closing connection Netty4TcpChannel{localAddress=/10.1.1.6:9300, remoteAddress=/192.168.70.128:40156, profile=default}", "elasticsearch.cluster.name":"cluster2"}

The internal IP addresses on VM 1 and VM 2 for the containers are 10.1.1.2 and 10.1.1.6, respectively. I have set a separate local IP for each container. The issue occurs when trying to connect from the node on Cluster 1 (with IP 10.1.1.2) to the node on Cluster 2 (with IP 10.1.1.6).

Here is a snippet of the Docker Compose configuration for the node on VM 1:

image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
container_name: elasticsearch
volumes:

  • certs:/usr/share/elasticsearch/config/certs
  • esdata01:/usr/share/elasticsearch/data
    ports:
  • 9200:9200
  • 9300:9300
    environment:
  • node.name=elasticsearch
  • cluster.name=${CLUSTER_NAME}
  • ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
  • bootstrap.memory_lock=true
  • discovery.type=single-node
  • xpack.security.enabled=true
  • xpack.security.http.ssl.enabled=true
  • xpack.security.http.ssl.key=certs/elasticsearch/elasticsearch.key
  • xpack.security.http.ssl.certificate=certs/elasticsearch/elasticsearch.crt
  • xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
  • xpack.security.transport.ssl.enabled=true
  • xpack.security.transport.ssl.key=certs/elasticsearch/elasticsearch.key
  • xpack.security.transport.ssl.certificate=certs/elasticsearch/elasticsearch.crt
  • xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
  • xpack.security.transport.ssl.verification_mode=certificate
  • network.host=0.0.0.0
  • transport.host=0.0.0.0
    mem_limit: ${MEM_LIMIT}
    ulimits:
    memlock:
    soft: -1
    hard: -1
    networks:
    esnet:
    ipv4_address: 10.1.1.2

Node 1 configuration on VM1 is set with local IP 10.1.1.2, and Node 2 on VM2 has IP 10.1.1.6. However, I am facing routing problems, and the nodes are still unable to connect to each other.

I would greatly appreciate your support in resolving this issue and helping the nodes on different VMs connect with each other successfully.

Thank you!