Error message": "getting node from external cluster [RC_Test] failed" between nodes on 2 docker hosts

Elastic Stack Elasticsearch
1

Hi,

We are facing problem while connecting remote cluster. I am using docker compose to build elasticsearch with local ip 10.1.1.2 on docker host 1 and same on docker host 2, when trying to connect remote cluster it is giving error because we are using local ip of container to communicate with each other.

{"@timestamp":"2025-02-18T03:48:51.727Z", "log.level": "WARN", "message":"fetching nodes from external cluster [RC_Test] failed", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch][generic][T#4]","log.logger":"org.elasticsearch.transport.SniffConnectionStrategy","elasticsearch.cluster.uuid":"xpKA78N7Rv-3McOx06AMOQ","elasticsearch.node.id":"jedJe4hIShud73LYpFj93w","elasticsearch.node.name":"elasticsearch","elasticsearch.cluster.name":"cluster","error.type":"org.elasticsearch.transport.ConnectTransportException","error.message":"[elasticsearch][10.1.1.2:9300] handshake failed. unexpected remote node {elasticsearch}{jedJe4hIShud73LYpFj93w}{qxEjhNNyR4KmMEmQturMuQ}{elasticsearch}{10.1.1.2}{10.1.1.2:9300}{dimr}{8.12.2}{7000099-8500010}{ml.config_version=12.0.0, xpack.installed=true,

2

Are you running both nodes on two different machines within the network or running two different containers within the same machines.

Can you provide more details about this?

3

Hi, I run 2 different VMWare in the network, each VM has 3 container nodes.

4

I am confused with container nodes. Can you elaborate with details.

5

On each VM there are three elasticsearch nodes forming a cluster and they have been communicating with each other on their cluster. I am getting an error when remoting cluster 1 to cluster 2. But when connecting to the remote cluster, the node on cluster 1 and the node on cluster 2 use the internal ip of the node (ip assigned to each container) to communicate with each other. And the internal ip of docker assigned to each container on the 2 clusters is different and cannot be on the same network.

6

Can you check the network.host is set to 0.0.0.0. it allows you to connect from the remote machine. Run the containers on host network.

7

Thank you, please support me again!
i added network=0.0.0.0 but it still seems to try to use the node's local IP address.
log elasticsearch on cluster 1: {"@timestamp":"2025-02-25T10:18:47.111Z", "log.level": "INFO", "message":"updating [cluster.remote.RC.mode] from [SNIFF] to [sniff]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.common.settings.ClusterSettings","elasticsearch.cluster.uuid":"owWXPNzxQkmlPCKXD2KAmw","elasticsearch.node.id":"aK5N84e3TBukArEY9GfZ5A","elasticsearch.node.name":"elasticsearch","elasticsearch.cluster.name":"cluster1"}
{"@timestamp":"2025-02-25T10:18:47.112Z", "log.level": "INFO", "message":"updating [cluster.remote.RC.seeds] from [] to [["192.168.70.129:9300"]]", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch][clusterApplierService#updateTask][T#1]","log.logger":"org.elasticsearch.common.settings.ClusterSettings","elasticsearch.cluster.uuid":"owWXPNzxQkmlPCKXD2KAmw","elasticsearch.node.id":"aK5N84e3TBukArEY9GfZ5A","elasticsearch.node.name":"elasticsearch","elasticsearch.cluster.name":"cluster1"}
{"@timestamp":"2025-02-25T10:18:50.543Z", "log.level": "WARN", "message":"fetching nodes from external cluster [RC] failed", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch][generic][T#5]","log.logger":"org.elasticsearch.transport.SniffConnectionStrategy","elasticsearch.cluster.uuid":"owWXPNzxQkmlPCKXD2KAmw","elasticsearch.node.id":"aK5N84e3TBukArEY9GfZ5A","elasticsearch.node.name":"elasticsearch","elasticsearch.cluster.name":"cluster1","error.type":"org.elasticsearch.transport.ConnectTransportException","error.message":"[elasticsearch][10.1.1.6:9300] connect_exception","error.stack_trace":"org.elasticsearch.transport.ConnectTransportException: [elasticsearch][10.1.1.6:9300] connect_exception\n\tat org.elasticsearch.server@8.17.2/org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onFailure(TcpTransport.java:1145)\n\tat org.elasticsearch.server@8.17.2/org.elasticsearch.action.support.SubscribableListener$FailureResult.complete(SubscribableListener.java:399)\n\tat org.elasticsearch.server@8.17.2/org.elasticsearch.action.support.SubscribableListener.tryComplete(SubscribableListener.java:311)\n\tat org.elasticsearch.server@8.17.2/org.elasticsearch.action.support.SubscribableListener.setResult(SubscribableListener.java:336)\n\tat org.elasticsearch.server@8.17.2/org.elasticsearch.action.support.SubscribableListener.onFailure(SubscribableListener.java:255)\n\tat org.elasticsearch.transport.netty4@8.17.2/org.elasticsearch.transport.netty4.Netty4Utils.lambda$addListener$2(Netty4Utils.java:237)\n\tat io.netty.common@4.1.115.Final/io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590)\n\tat io.netty.common@4.1.115.Final/io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583)\n\tat io.netty.common@4.1.115.Final/io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559)\n\tat io.netty.common@4.1.115.Final/io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492)\n\tat io.netty.common@4.1.115.Final/io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636)\n\tat io.netty.common@4.1.115.Final/io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629)\n\tat io.netty.common@4.1.115.Final/io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118)\n\tat org.elasticsearch.security@8.17.2/org.elasticsearch.xpack.core.security.transport.netty4.SecurityNetty4Transport$ClientSslHandlerInitializer.lambda$connect$1(SecurityNetty4Transport.java:347)\n\tat io.netty.common@4.1.115.Final/io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590)\n\tat io.netty.common@4.1.115.Final/io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583)\n\tat io.netty.common@4.1.115.Final/io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559)\n\tat io.netty.common@4.1.115.Final/io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492)\n\tat io.netty.common@4.1.115.Final/io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636)\n\tat io.netty.common@4.1.115.Final/io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:629)\n\tat io.netty.common@4.1.115.Final/io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:118)\n\tat io.netty.transport@4.1.115.Final/io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:326)\n\tat io.netty.transport@4.1.115.Final/io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:342)\n\tat io.netty.transport@4.1.115.Final/io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776)\n\tat io.netty.transport@4.1.115.Final/io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689)\n\tat io.netty.transport@4.1.115.Final/io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652)\n\tat io.netty.transport@4.1.115.Final/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)\n\tat io.netty.common@4.1.115.Final/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)\n\tat io.netty.common@4.1.115.Final/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat java.base/java.lang.Thread.run(Thread.java:1575)\nCaused by: org.elasticsearch.common.util.concurrent.UncategorizedExecutionException: Failed execution\n\tat org.elasticsearch.server@8.17.2/org.elasticsearch.action.support.SubscribableListener.wrapAsExecutionException(SubscribableListener.java:292)\n\tat org.elasticsearch.server@8.17.2/org.elasticsearch.common.util.concurrent.ListenableFuture.wrapException(ListenableFuture.java:39)\n\tat org.elasticsearch.server@8.17.2/org.elasticsearch.common.util.concurrent.ListenableFuture.wrapException(ListenableFuture.java:28)\n\t... 26 more\nCaused by: java.util.concurrent.ExecutionException: io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: No route to host: 10.1.1.6/10.1.1.6:9300\n\t... 29 more\nCaused by: io.netty.channel.AbstractChannel$AnnotatedNoRouteToHostException: No route to host: 10.1.1.6/10.1.1.6:9300\nCaused by: java.net.NoRouteToHostException: No route to host\n\tat java.base/sun.nio.ch.Net.pollConnect(Native Method)\n\tat java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)\n\tat java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:1062)\n\tat io.netty.transport@4.1.115.Final/io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:336)\n\tat io.netty.transport@4.1.115.Final/io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:339)\n\tat io.netty.transport@4.1.115.Final/io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:776)\n\tat io.netty.transport@4.1.115.Final/io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689)\n\tat io.netty.transport@4.1.115.Final/io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652)\n\tat io.netty.transport@4.1.115.Final/io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)\n\tat io.netty.common@4.1.115.Final/io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)\n\tat io.netty.common@4.1.115.Final/io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\n\tat java.base/java.lang.Thread.run(Thread.java:1575)\n"}

log elasticsaerch on cluster 2: {"@timestamp":"2025-02-25T10:19:27.098Z", "log.level": "WARN", "message":"received plaintext traffic on an encrypted channel, closing connection Netty4TcpChannel{localAddress=/10.1.1.6:9300, remoteAddress=/192.168.70.128:40156, profile=default}", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[elasticsearch][transport_worker][T#2]","log.logger":"org.elasticsearch.xpack.core.security.transport.netty4.SecurityNetty4Transport","elasticsearch.cluster.uuid":"gA64FW4xTLC7VBq3yMeJSg","elasticsearch.node.id":"AUSvtg5oTNWscMvcIGSShg","elasticsearch.node.name":"elasticsearch","elasticsearch.cluster.name":"cluster2"}

IP address 10.1.1.6 is elasticsearch node on cluster 2

8

I see nodes on 2 VMs are having routing issues. Is there a way for 2 nodes on different VMs to communicate with each other?

9

Are you running containers on host network?

Can you also set the following and see

- network.host=0.0.0.0
- transport.host=0.0.0.0

Do you have ports opened between those machines ?

Is it possible to share your configuration?

10

No, I set separate local IP for each container.

image: docker.elastic.co/elasticsearch/elasticsearch:${STACK_VERSION}
container_name: elasticsearch
volumes:
- certs:/usr/share/elasticsearch/config/certs
- esdata01:/usr/share/elasticsearch/data
ports:
- 9200:9200
- 9300:9300
environment:
- node.name=elasticsearch
- cluster.name=${CLUSTER_NAME}
- ELASTIC_PASSWORD=${ELASTIC_PASSWORD}
- bootstrap.memory_lock=true
- discovery.type=single-node
- xpack.security.enabled=true
- xpack.security.http.ssl.enabled=true
- xpack.security.http.ssl.key=certs/elasticsearch/elasticsearch.key
- xpack.security.http.ssl.certificate=certs/elasticsearch/elasticsearch.crt
- xpack.security.http.ssl.certificate_authorities=certs/ca/ca.crt
- xpack.security.transport.ssl.enabled=true
- xpack.security.transport.ssl.key=certs/elasticsearch/elasticsearch.key
- xpack.security.transport.ssl.certificate=certs/elasticsearch/elasticsearch.crt
- xpack.security.transport.ssl.certificate_authorities=certs/ca/ca.crt
- xpack.security.transport.ssl.verification_mode=certificate
- xpack.license.self_generated.type=${LICENSE}
- network.host=0.0.0.0
- transport.host=0.0.0.0
mem_limit: ${MEM_LIMIT}
ulimits:
memlock:
soft: -1
hard: -1
networks:
esnet:
ipv4_address: 10.1.1.2

This is the node one configuration on VM one with local node ip 10.1.1.2 and ip: 10.1.1.6 on VM two

11 
- network.host=0.0.0.0
- transport.host=0.0.0.0
- network.publish_host=<SERVER_IP>

Can you try setting these settings.

Remote Cluster Settings: On the cluster that connects to this node, make sure to use the host IP:
cluster.remote.RC.seeds: ["192.168.70.129:9300"]