# Error message": "getting node from external cluster \[RC\_Test\] failed" between nodes on 2 docker hosts **URL:** \ **Category:** Elasticsearch\ **Tags:** docker\ **Created:** [February 25, 2025, 2:57am UTC](https://discuss.elastic.co/t/error-message-getting-node-from-external-cluster-rc-test-failed-between-nodes-on-2-docker-hosts/374980 "2025-02-25T02:57:40Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![poke1710](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/poke1710/32/141609_2.png) [@poke1710](https://discuss.elastic.co/u/poke1710)\ **Post date:** [February 25, 2025, 2:57am UTC](https://discuss.elastic.co/t/error-message-getting-node-from-external-cluster-rc-test-failed-between-nodes-on-2-docker-hosts/374980/1 "2025-02-25T02:57:40Z")
Hi, We are facing problem while connecting remote cluster. I am using docker compose to build elasticsearch with local ip 10.1.1.2 on docker host 1 and same on docker host 2, when trying to connect remote cluster it is giving error because we are using local ip of container to communicate with each other. {"@timestamp":"2025-02-18T03:48:51.727Z", "log.level": "WARN", "message":"fetching nodes from external cluster [RC\_Test] failed", "ecs.version": "1.2.0","service.name":"ES\_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch][generic][T#4]","log.logger":"org.elasticsearch.transport.SniffConnectionStrategy","elasticsearch.cluster.uuid":"xpKA78N7Rv-3McOx06AMOQ","elasticsearch.node.id":"jedJe4hIShud73LYpFj93w","elasticsearch.node.name":"elasticsearch","elasticsearch.cluster.name":"cluster","error.type":"org.elasticsearch.transport.ConnectTransportException","error.message":"[elasticsearch][10.1.1.2:9300] handshake failed. unexpected remote node {elasticsearch}{jedJe4hIShud73LYpFj93w}{qxEjhNNyR4KmMEmQturMuQ}{elasticsearch}{10.1.1.2}{10.1.1.2:9300}{dimr}{8.12.2}{7000099-8500010}{ml.config\_version=12.0.0, xpack.installed=true, ---
**Author:** ![JinnaBalu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jinnabalu/32/27899_2.png) [@JinnaBalu](https://discuss.elastic.co/u/JinnaBalu)\ **Post date:** [February 25, 2025, 6:12am UTC](https://discuss.elastic.co/t/error-message-getting-node-from-external-cluster-rc-test-failed-between-nodes-on-2-docker-hosts/374980/2 "2025-02-25T06:12:59Z")
Are you running both nodes on two different machines within the network or running two different containers within the same machines. Can you provide more details about this? ---
**Author:** ![poke1710](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/poke1710/32/141609_2.png) [@poke1710](https://discuss.elastic.co/u/poke1710)\ **Post date:** [February 25, 2025, 6:48am UTC](https://discuss.elastic.co/t/error-message-getting-node-from-external-cluster-rc-test-failed-between-nodes-on-2-docker-hosts/374980/3 "2025-02-25T06:48:59Z")
Hi, I run 2 different VMWare in the network, each VM has 3 container nodes. ---
**Author:** ![JinnaBalu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jinnabalu/32/27899_2.png) [@JinnaBalu](https://discuss.elastic.co/u/JinnaBalu)\ **Post date:** [February 25, 2025, 7:10am UTC](https://discuss.elastic.co/t/error-message-getting-node-from-external-cluster-rc-test-failed-between-nodes-on-2-docker-hosts/374980/4 "2025-02-25T07:10:04Z")
I am confused with container nodes. Can you elaborate with details. ---
**Author:** ![poke1710](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/poke1710/32/141609_2.png) [@poke1710](https://discuss.elastic.co/u/poke1710)\ **Post date:** [February 25, 2025, 7:18am UTC](https://discuss.elastic.co/t/error-message-getting-node-from-external-cluster-rc-test-failed-between-nodes-on-2-docker-hosts/374980/5 "2025-02-25T07:18:14Z")
On each VM there are three elasticsearch nodes forming a cluster and they have been communicating with each other on their cluster. I am getting an error when remoting cluster 1 to cluster 2. But when connecting to the remote cluster, the node on cluster 1 and the node on cluster 2 use the internal ip of the node (ip assigned to each container) to communicate with each other. And the internal ip of docker assigned to each container on the 2 clusters is different and cannot be on the same network. ---
**Author:** ![JinnaBalu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jinnabalu/32/27899_2.png) [@JinnaBalu](https://discuss.elastic.co/u/JinnaBalu)\ **Post date:** [February 25, 2025, 7:38am UTC](https://discuss.elastic.co/t/error-message-getting-node-from-external-cluster-rc-test-failed-between-nodes-on-2-docker-hosts/374980/6 "2025-02-25T07:38:30Z")
Can you check the `network.host` is set to `0.0.0.0`. it allows you to connect from the remote machine. Run the containers on host network. ---
**Author:** ![poke1710](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/poke1710/32/141609_2.png) [@poke1710](https://discuss.elastic.co/u/poke1710)\ **Post date:** [February 25, 2025, 10:30am UTC](https://discuss.elastic.co/t/error-message-getting-node-from-external-cluster-rc-test-failed-between-nodes-on-2-docker-hosts/374980/7 "2025-02-25T10:30:32Z")
Thank you, please support me again! i added network=0.0.0.0 but it still seems to try to use the node's local IP address. log elasticsearch on cluster 1: {"@timestamp":"2025-02-25T10:18:47.111Z", "log.level": "INFO", "message":"updating [cluster.remote.RC.mode] from [SNIFF] to [sniff]", "ecs.version": "1.2.0","service.name":"ES\_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.common.settings.ClusterSettings","elasticsearch.cluster.uuid":"owWXPNzxQkmlPCKXD2KAmw","elasticsearch.node.id":"aK5N84e3TBukArEY9GfZ5A","elasticsearch.node.name":"elasticsearch","elasticsearch.cluster.name":"cluster1"} {"@timestamp":"2025-02-25T10:18:47.112Z", "log.level": "INFO", "message":"updating [cluster.remote.RC.seeds] from [] to [["192.168.70.129:9300"]]", "ecs.version": "1.2.0","service.name":"ES\_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.common.settings.ClusterSettings","elasticsearch.cluster.uuid":"owWXPNzxQkmlPCKXD2KAmw","elasticsearch.node.id":"aK5N84e3TBukArEY9GfZ5A","elasticsearch.node.name":"elasticsearch","elasticsearch.cluster.name":"cluster1"} {"@timestamp":"2025-02-25T10:18:50.543Z", "log.level": "WARN", "message":"fetching nodes from external cluster [RC] failed", "ecs.version": "1.2.0","service.name":"ES\_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch][generic][T#5]","log.logger":"org.elasticsearch.transport.SniffConnectionStrategy","elasticsearch.cluster.uuid":"owWXPNzxQkmlPCKXD2KAmw","elasticsearch.node.id":"aK5N84e3TBukArEY9GfZ5A","elasticsearch.node.name":"elasticsearch","elasticsearch.cluster.name":"cluster1","error.type":"org.elasticsearch.transport.ConnectTransportException","error.message":"[elasticsearch][10.1.1.6:9300] connect\_exception","error.stack\_trace":"org.elasticsearch.transport.ConnectTransportException: [elasticsearch][10.1.1.6:9300] connect\_exception\n\tat org.elasticsearch.server@8.17.2/org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1145)\n\tat org.elasticsearch.server@8.17.2/org.elasticsearch.action.support.SubscribableListener$FailureResult.complete(SubscribableListener.java:399)\n\tat org.elasticsearch.server@8.17.2/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:311)\n\tat org.elasticsearch.server@8.17.2/org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:336)\n\tat org.elasticsearch.server@8.17.2/org.elasticsearch.action.support.SubscribableListener.onFailure(SubscribableListener.java:255)\n\tat org.elasticsearch.transport.netty4@8.17.2/org.elasticsearch.transport.netty4.Netty4Utils.lambda$addListener$2(Netty4Utils.java:237)\n\tat io.netty.common@4.1.115.Final/io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590)\n\tat io.netty.common@4.1.115.Final/io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583)\n\tat io.netty.common@4.1.115.Final/io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559)\n\tat io.netty.common@4.1.115.Final/io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492)\n\tat io.netty.common@4.1.115.Final/io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636)\n\tat io.netty.common@4.1.115.Final/io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629)\n\tat io.netty.common@4.1.115.Final/io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118)\n\tat org.elasticsearch.security@8.17.2/org.elasticsearch.xpack.core.security.transport.netty4.SecurityNetty4Transport$ClientSslHandlerInitializer.lambda$connect$1(SecurityNetty4Transport.java:347)\n\tat io.netty.common@4.1.115.Final/io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590)\n\tat io.netty.common@4.1.115.Final/io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583)\n\tat io.netty.common@4.1.115.Final/io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559)\n\tat io.netty.common@4.1.115.Final/io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492)\n\tat io.netty.common@4.1.115.Final/io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636)\n\tat io.netty.common@4.1.115.Final/io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629)\n\tat io.netty.common@4.1.115.Final/io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118)\n\tat io.netty.transport@4.1.115.Final/io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:326)\n\tat io.netty.transport@4.1.115.Final/io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:342)\n\tat io.netty.transport@4.1.115.Final/io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776)\n\tat io.netty.transport@4.1.115.Final/io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689)\n\tat io.netty.transport@4.1.115.Final/io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652)\n\tat io.netty.transport@4.1.115.Final/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)\n\tat io.netty.common@4.1.115.Final/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)\n\tat io.netty.common@4.1.115.Final/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat java.base/java.lang.Thread.run(Thread.java:1575)\nCaused by: org.elasticsearch.common.util.concurrent.UncategorizedExecutionException: Failed execution\n\tat org.elasticsearch.server@8.17.2/org.elasticsearch.action.support.SubscribableListener.wrapAsExecutionException(SubscribableListener.java:292)\n\tat org.elasticsearch.server@8.17.2/org.elasticsearch.common.util.concurrent.ListenableFuture.wrapException(ListenableFuture.java:39)\n\tat org.elasticsearch.server@8.17.2/org.elasticsearch.common.util.concurrent.ListenableFuture.wrapException(ListenableFuture.java:28)\n\t... 26 more\nCaused by: java.util.concurrent.ExecutionException: io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: No route to host: 10.1.1.6/10.1.1.6:9300\n\t... 29 more\nCaused by: io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: No route to host: 10.1.1.6/10.1.1.6:9300\nCaused by: java.net.NoRouteToHostException: No route to host\n\tat java.base/sun.nio.ch.Net.pollConnect(Native Method)\n\tat java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)\n\tat java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:1062)\n\tat io.netty.transport@4.1.115.Final/io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:336)\n\tat io.netty.transport@4.1.115.Final/io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:339)\n\tat io.netty.transport@4.1.115.Final/io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776)\n\tat io.netty.transport@4.1.115.Final/io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689)\n\tat io.netty.transport@4.1.115.Final/io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652)\n\tat io.netty.transport@4.1.115.Final/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)\n\tat io.netty.common@4.1.115.Final/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)\n\tat io.netty.common@4.1.115.Final/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat java.base/java.lang.Thread.run(Thread.java:1575)\n"} log elasticsaerch on cluster 2: {"@timestamp":"2025-02-25T10:19:27.098Z", "log.level": "WARN", "message":"received plaintext traffic on an encrypted channel, closing connection Netty4TcpChannel{localAddress=/10.1.1.6:9300, remoteAddress=/192.168.70.128:40156, profile=default}", "ecs.version": "1.2.0","service.name":"ES\_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch][transport\_worker][T#2]","log.logger":"org.elasticsearch.xpack.core.security.transport.netty4.SecurityNetty4Transport","elasticsearch.cluster.uuid":"gA64FW4xTLC7VBq3yMeJSg","elasticsearch.node.id":"AUSvtg5oTNWscMvcIGSShg","elasticsearch.node.name":"elasticsearch","elasticsearch.cluster.name":"cluster2"} IP address 10.1.1.6 is elasticsearch node on cluster 2 ---
**Author:** ![poke1710](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/poke1710/32/141609_2.png) [@poke1710](https://discuss.elastic.co/u/poke1710)\ **Post date:** [February 25, 2025, 10:43am UTC](https://discuss.elastic.co/t/error-message-getting-node-from-external-cluster-rc-test-failed-between-nodes-on-2-docker-hosts/374980/8 "2025-02-25T10:43:44Z")
I see nodes on 2 VMs are having routing issues. Is there a way for 2 nodes on different VMs to communicate with each other? ---
**Author:** ![JinnaBalu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jinnabalu/32/27899_2.png) [@JinnaBalu](https://discuss.elastic.co/u/JinnaBalu)\ **Post date:** [February 25, 2025, 10:46am UTC](https://discuss.elastic.co/t/error-message-getting-node-from-external-cluster-rc-test-failed-between-nodes-on-2-docker-hosts/374980/9 "2025-02-25T10:46:21Z")
Are you running containers on host network? Can you also set the following and see ```auto - network.host=0.0.0.0 - transport.host=0.0.0.0 ``` Do you have ports opened between those machines ? Is it possible to share your configuration? ---
**Author:** ![poke1710](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/poke1710/32/141609_2.png) [@poke1710](https://discuss.elastic.co/u/poke1710)\ **Post date:** [February 25, 2025, 10:53am UTC](https://discuss.elastic.co/t/error-message-getting-node-from-external-cluster-rc-test-failed-between-nodes-on-2-docker-hosts/374980/10 "2025-02-25T10:53:21Z")
No, I set separate local IP for each container. image: [docker.elastic.co/elasticsearch/elasticsearch:${STACK\_VERSION}](http://docker.elastic.co/elasticsearch/elasticsearch:$%7BSTACK_VERSION%7D) container\_name: elasticsearch volumes: - certs:/usr/share/elasticsearch/config/certs - esdata01:/usr/share/elasticsearch/data ports: - 9200:9200 - 9300:9300 environment: - node.name=elasticsearch - cluster.name=${CLUSTER\_NAME} - ELASTIC\_PASSWORD=${ELASTIC\_PASSWORD} - bootstrap.memory\_lock=true - discovery.type=single-node - xpack.security.enabled=true - xpack.security.http.ssl.enabled=true - xpack.security.http.ssl.key=certs/elasticsearch/elasticsearch.key - xpack.security.http.ssl.certificate=certs/elasticsearch/elasticsearch.crt - xpack.security.http.ssl.certificate\_authorities=certs/ca/ca.crt - xpack.security.transport.ssl.enabled=true - xpack.security.transport.ssl.key=certs/elasticsearch/elasticsearch.key - xpack.security.transport.ssl.certificate=certs/elasticsearch/elasticsearch.crt - xpack.security.transport.ssl.certificate\_authorities=certs/ca/ca.crt - xpack.security.transport.ssl.verification\_mode=certificate - xpack.license.self\_generated.type=${LICENSE} - network.host=0.0.0.0 - transport.host=0.0.0.0 mem\_limit: ${MEM\_LIMIT} ulimits: memlock: soft: -1 hard: -1 networks: esnet: ipv4\_address: 10.1.1.2 This is the node one configuration on VM one with local node ip 10.1.1.2 and ip: 10.1.1.6 on VM two ---
**Author:** ![JinnaBalu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jinnabalu/32/27899_2.png) [@JinnaBalu](https://discuss.elastic.co/u/JinnaBalu)\ **Post date:** [February 25, 2025, 4:21pm UTC](https://discuss.elastic.co/t/error-message-getting-node-from-external-cluster-rc-test-failed-between-nodes-on-2-docker-hosts/374980/11 "2025-02-25T16:21:15Z")
```auto - network.host=0.0.0.0 - transport.host=0.0.0.0 - network.publish_host= ``` Can you try setting these settings. **Remote Cluster Settings:** On the cluster that connects to this node, make sure to use the host IP: `cluster.remote.RC.seeds: ["192.168.70.129:9300"]`