# TWO Clusters unable to communicate for remote search **URL:** \ **Category:** Elasticsearch\ **Created:** [March 14, 2019, 9:54am UTC](https://discuss.elastic.co/t/two-clusters-unable-to-communicate-for-remote-search/172299 "2019-03-14T09:54:51Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![srirama](https://avatars.discourse-cdn.com/v4/letter/s/54ee81/32.png) [@srirama](https://discuss.elastic.co/u/srirama)\ **Post date:** [March 14, 2019, 9:54am UTC](https://discuss.elastic.co/t/two-clusters-unable-to-communicate-for-remote-search/172299/1 "2019-03-14T09:54:52Z")
In my local development servers, we created two clusters, each having single node. The cluster status for each of them made green by making number of replicas to 0. Below is the cluster configurations. ```auto { "persistent": { "search": { "remote": { "new-cluster": { "seeds": [ "172.25.24.160:9300" ] }, "nc3": { "seeds": [ "172.25.25.237:9300" ] } } } }, "transient": {} } ``` When checked the network level settings there is no issue observed. When the send the search query using kibana tool, remote cluster search is working normally. However when search request sent via the REST API it is saying unable to communicate with remote cluster as below. ```auto org.elasticsearch.transport.TransportException: unable to communicate with remote cluster [new-cluster] at org.elasticsearch.action.search.RemoteClusterService$1.onFailure(RemoteClusterService.java:286) ~[elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:67) ~[elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.action.ActionListener.onFailure(ActionListener.java:101) ~[elasticsearch-5.4.0.jar:5.4.0] ``` Below is the logs at the elastic search startup time, ```auto [2019-03-14T15:40:48,618][WARN][o.e.a.s.RemoteClusterService] [node-3] failed to update seed list for cluster: new-cluster org.elasticsearch.transport.ConnectTransportException: [node-1][10.0.30.48:9300] connect_timeout[30s] at org.elasticsearch.transport.netty4.Netty4Transport.connectToChannels(Netty4Transport.java:359) ~[?:?] at org.elasticsearch.transport.TcpTransport.openConnection(TcpTransport.java:526) ~[elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.transport.TcpTransport.connectToNode(TcpTransport.java:465) ~[elasticsearch-5.4.0.jar:5.4.0] at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:315) ~[elasticsearch-5.4.0.jar:5.4.0] ``` Here the `10.0.30.48:9300` is the internal IP address. ELK is running in the cloud-servers. In the elasticsearch.yml file network.host is given as below. ```auto network.host: 10.0.30.48 ``` My elastic search version is 5.4.0 on both clusters. Below is curl command output. ```auto curl 172.25.24.160:9200 { "name" : "node-1", "cluster_name" : "new-cluster", "cluster_uuid" : "CWn8aXerToK2bc1O4VBrjw", "version" : { "number" : "5.4.0", "build_hash" : "780f8c4", "build_date" : "2017-04-28T17:43:27.229Z", "build_snapshot" : false, "lucene_version" : "6.5.0" }, "tagline" : "You Know, for Search" } ``` ```auto curl 172.25.25.237:9200 { "name" : "node-3", "cluster_name" : "nc3", "cluster_uuid" : "28hSUEv8T3aSS5MN16sZVg", "version" : { "number" : "5.4.0", "build_hash" : "780f8c4", "build_date" : "2017-04-28T17:43:27.229Z", "build_snapshot" : false, "lucene_version" : "6.5.0" }, "tagline" : "You Know, for Search" } ``` The search request URI is same in both the cases. ---
**Author:** ![HenningAndersen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/henningandersen/32/48188_2.png) [@HenningAndersen](https://discuss.elastic.co/u/HenningAndersen)\ **Post date:** [March 14, 2019, 1:16pm UTC](https://discuss.elastic.co/t/two-clusters-unable-to-communicate-for-remote-search/172299/2 "2019-03-14T13:16:43Z")
For this to work, I believe the two clusters must be able to reach each other on the ip address listed in network.host. Does it work if you use the public IP instead? ---
**Author:** ![srirama](https://avatars.discourse-cdn.com/v4/letter/s/54ee81/32.png) [@srirama](https://discuss.elastic.co/u/srirama)\ **Post date:** [March 14, 2019, 1:37pm UTC](https://discuss.elastic.co/t/two-clusters-unable-to-communicate-for-remote-search/172299/3 "2019-03-14T13:37:39Z")
Thanks for reply. Yes these ELK clusters are running behind the firewall/proxy. The IP address is getting translated to internal IP. The CURL command on public IP is working and getting o/p as shown earlier. Here ```auto cluster name Internal IP External IP network.host in yml file nc3 10.0.21.92 172.25.25.237 10.0.21.92 new-cluster 10.0.30.48 172.25.24.160 10.0.30.48 ``` Here curl command on public IP and output is shared. If I kept the public IP in network.host then ELK is not coming up as it is not able to bind on that IP. ---
**Author:** ![HenningAndersen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/henningandersen/32/48188_2.png) [@HenningAndersen](https://discuss.elastic.co/u/HenningAndersen)\ **Post date:** [March 14, 2019, 2:47pm UTC](https://discuss.elastic.co/t/two-clusters-unable-to-communicate-for-remote-search/172299/4 "2019-03-14T14:47:14Z")
Hi @srirama, is port 9300 open between the clusters? You should be able to test that by logging onto one of the nodes and doing telnet ip 9300 against the other nodes ip. This should succeed, otherwise, they will not be able to communicate (and thus do cross cluster search). ---
**Author:** ![srirama](https://avatars.discourse-cdn.com/v4/letter/s/54ee81/32.png) [@srirama](https://discuss.elastic.co/u/srirama)\ **Post date:** [March 15, 2019, 5:28am UTC](https://discuss.elastic.co/t/two-clusters-unable-to-communicate-for-remote-search/172299/5 "2019-03-15T05:28:38Z")
Hi @HenningAndersen Yes these nodes are reachable to each other on public ip on the ports 9200 and 9300. Verified the telnet. ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/9/b9cd8551da2c7c2dfd4d196b64af4442b69e11e1.png) ---
**Author:** ![HenningAndersen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/henningandersen/32/48188_2.png) [@HenningAndersen](https://discuss.elastic.co/u/HenningAndersen)\ **Post date:** [March 15, 2019, 7:42am UTC](https://discuss.elastic.co/t/two-clusters-unable-to-communicate-for-remote-search/172299/6 "2019-03-15T07:42:54Z")
Hi @srirama, thanks for providing this. Can we repeat the telnet exercise against the internal ip addresses too? ---
**Author:** ![srirama](https://avatars.discourse-cdn.com/v4/letter/s/54ee81/32.png) [@srirama](https://discuss.elastic.co/u/srirama)\ **Post date:** [March 15, 2019, 9:19am UTC](https://discuss.elastic.co/t/two-clusters-unable-to-communicate-for-remote-search/172299/7 "2019-03-15T09:19:23Z")
Hi @HenningAndersen: On the internal IP, they are not accessible to each other. Only using Public IP accessible. ---
**Author:** ![HenningAndersen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/henningandersen/32/48188_2.png) [@HenningAndersen](https://discuss.elastic.co/u/HenningAndersen)\ **Post date:** [March 15, 2019, 9:25am UTC](https://discuss.elastic.co/t/two-clusters-unable-to-communicate-for-remote-search/172299/8 "2019-03-15T09:25:10Z")
HI @srirama, OK. For a development setup, you could probably fix this by using the public IP address as the publish\_host on each cluster, ie. set: `network.publish_host=172.25.25.237` on `nc3` `network.publish_host=172.25.24.160` on `new-cluster` ---
**Author:** ![srirama](https://avatars.discourse-cdn.com/v4/letter/s/54ee81/32.png) [@srirama](https://discuss.elastic.co/u/srirama)\ **Post date:** [March 15, 2019, 9:40am UTC](https://discuss.elastic.co/t/two-clusters-unable-to-communicate-for-remote-search/172299/9 "2019-03-15T09:40:36Z")
Hi @HenningAndersen: Yes it is working now. After changing the configuration I restarted the ELK at that time, which ever cluster node is restarted last it is binding to other. But at the same time the in other node remote info, seeds are becoming empty. Why is this so? So I updated the cluster settings again in the node where seeds become empty, then the clusters are joined. Now both are up and joined. Also if internal IPs are reachable to each other then what setting need to be changed. In the cluster definition need to provide internal IPs alone or any more settings need to be tuned. Thanks for information. ---
**Author:** ![srirama](https://avatars.discourse-cdn.com/v4/letter/s/54ee81/32.png) [@srirama](https://discuss.elastic.co/u/srirama)\ **Post date:** [March 15, 2019, 11:29am UTC](https://discuss.elastic.co/t/two-clusters-unable-to-communicate-for-remote-search/172299/10 "2019-03-15T11:29:04Z")
Pl. check the [https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html#advanced-network-settings](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-network.html#advanced-network-settings) for the explanation why the above setting is suggested. Also in case of internal IPs are reachable to each other other than specifying `network.host` no other configuration changes are required. @HenningAndersen pl. correct me in case of wrong. ---
**Author:** ![HenningAndersen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/henningandersen/32/48188_2.png) [@HenningAndersen](https://discuss.elastic.co/u/HenningAndersen)\ **Post date:** [March 15, 2019, 11:40am UTC](https://discuss.elastic.co/t/two-clusters-unable-to-communicate-for-remote-search/172299/11 "2019-03-15T11:40:53Z")
HI @srirama, yes, AFAIK that is correct. If the nodes can reach each other on the internal IPs, you should be able to remove the `network.publish_host` setting. Figuring out why the remote seeds disappeared will take some more digging I think, preferably including reproducing this on your end with more precise description of steps. Do you find it important to get clarified? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 12, 2019, 11:40am UTC](https://discuss.elastic.co/t/two-clusters-unable-to-communicate-for-remote-search/172299/12 "2019-04-12T11:40:54Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.