# Cross Cluster Search gives strange results when remote nodes become unavailable

**URL:** <https://discuss.elastic.co/t/cross-cluster-search-gives-strange-results-when-remote-nodes-become-unavailable/159616>\
**Category:** Elasticsearch\
**Created:** [December 5, 2018, 10:44pm UTC](https://discuss.elastic.co/t/cross-cluster-search-gives-strange-results-when-remote-nodes-become-unavailable/159616 "2018-12-05T22:44:27Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![nma](https://avatars.discourse-cdn.com/v4/letter/n/b5e925/32.png) [@nma](https://discuss.elastic.co/u/nma)\
**Post date:** [December 5, 2018, 10:44pm UTC](https://discuss.elastic.co/t/cross-cluster-search-gives-strange-results-when-remote-nodes-become-unavailable/159616/1 "2018-12-05T22:44:27Z")

</div>

Hi.

**The problem**  
I've configured elasticsearch for Cross Cluster Search (CCS). It works well in most cases. But I can't get a correct behaviour in a situation when a remote node becomes unavailable [by pulling a network cable off]: CCS requests start hanging up.  
The expected behaviour is to get some results from available CCS nodes.

The CCS simplified deployment scheme  
node1/cluster1[1 shard]/index1  
node2/cluster2[1 shard]/index1

The desired behaviour  
node1 can do CCS requests to index1 of node1 and node2  
node2 can do CCS requests to index1 of node1 and node2

**Interesting behaviour particular qualities**  
Case 1

1. node1, node2: ES are working
2. node2: pull the network cable off
3. node1: the CCS query to node2:index1 is hanging up

Case 2

1. node2: ES is working, the node is not available (by the cable) for node1
2. node1: ES is restated
3. node1: make a CCS query to node2:index1 and get a result (w/o node2 data)
4. node2: insert the network cable
5. node1: make a CCS query to node2:index1 and get a result (w/ node2 data)
6. node2: pull the network cable off
7. node1: the CCS query to node2:index1 is hanging up

**The used configuration**  
node1.elasticsearch.yml

> cluster.name: cluster1  
> node.name: cluster1-node1  
> network.bind\_host: ["127.0.0.1", "\<Node1\_IPv4\>"]  
> network.publish\_host: ["\<Node1\_IPv4\>"]

node1.cluster.settings

> ```
> {"persistent": { "search": { "remote": {
> "node1": {
> "skip_unavailable": "true",
> "seeds": ["127.0.0.1:9300"]
> },
> "node2": {
> "skip_unavailable": "true",
> "seeds": ["<Node2_IPv4>:9300"]
> }
> }}}}
> 
> ```

The node2 settings are similar.

**Environment**  
Linux  
ES 6.2.2  
Java 9.X

Thank you in advance.

---

<div class="post-metadata">

**Author:** ![javanna](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/javanna/32/4698_2.png) [@javanna](https://discuss.elastic.co/u/javanna)\
**Post date:** [December 7, 2018, 2:02pm UTC](https://discuss.elastic.co/t/cross-cluster-search-gives-strange-results-when-remote-nodes-become-unavailable/159616/2 "2018-12-07T14:02:57Z")

</div>

Hi Maxim,  
see [Elasticsearch CCS: client get timeout when remote cluster is isolated by firewall](https://discuss.elastic.co/t/elasticsearch-ccs-client-get-timeout-when-remote-cluster-is-isolated-by-firewall/152019/8) for a similar recent discussion. Elasticsearch is currently not super quick detecting when remote clusters are down and that may make requests hang. We have been working on improving that and some changes in that direction will go out with the 6.6 version.

One thing you could do on your end is tweak OS settings like suggested by David in the linked discussion, see also [https://github.com/elastic/elasticsearch/issues/34405#issuecomment-429465972](https://github.com/elastic/elasticsearch/issues/34405#issuecomment-429465972) .

Cheers  
Luca

---

<div class="post-metadata">

**Author:** ![nma](https://avatars.discourse-cdn.com/v4/letter/n/b5e925/32.png) [@nma](https://discuss.elastic.co/u/nma)\
**Post date:** [December 7, 2018, 3:32pm UTC](https://discuss.elastic.co/t/cross-cluster-search-gives-strange-results-when-remote-nodes-become-unavailable/159616/3 "2018-12-07T15:32:34Z")

</div>

Ok.  
Thank you, Luca.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 4, 2019, 3:32pm UTC](https://discuss.elastic.co/t/cross-cluster-search-gives-strange-results-when-remote-nodes-become-unavailable/159616/4 "2019-01-04T15:32:35Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.