# Cross Cluster search, switching to an empty backup cluster

**URL:** <https://discuss.elastic.co/t/cross-cluster-search-switching-to-an-empty-backup-cluster/274601>\
**Category:** Elasticsearch\
**Tags:** ccs-cross-cluster-search\
**Created:** [June 1, 2021, 9:04am UTC](https://discuss.elastic.co/t/cross-cluster-search-switching-to-an-empty-backup-cluster/274601 "2021-06-01T09:04:20Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![driveirk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/driveirk/32/102253_2.png) [@driveirk](https://discuss.elastic.co/u/driveirk)\
**Post date:** [June 1, 2021, 9:04am UTC](https://discuss.elastic.co/t/cross-cluster-search-switching-to-an-empty-backup-cluster/274601/1 "2021-06-01T09:04:20Z")

</div>

Hello,  
I want to make a backup cluster for Cross Cluster search.  
I selected proxy\_mod, I use Haproxy as a proxy

Haproxy configuration example:

```auto
server work 1.1.1.1:9300 check on-marked-up shutdown-backup-sessions on-marked-down shutdown-sessions
server backup 2.2.2.2:9300 check backup on-marked-up shutdown-backup-sessions on-marked-down shutdown-sessions

```

When cluster "work" is unavailable, it switches to the backup server,  
But with this in elastic, I get the error:

```auto
    {
       "error": {
         "root_cause": [
           {
             "type": "illegal_state_exception",
             "reason": "Unable to open any proxy connections to remote cluster [lab]"
           }
         ],
         "type": "illegal_state_exception",
         "reason": "Unable to open any proxy connections to remote cluster [lab]"
       },
       "status": 500
    }

```

Restarting elasticity solves the problem, but if the traffic is switched back, the problem is reproduced again.

Importantly, the "work" and "backup" clusters are not related in any way.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [June 2, 2021, 2:28am UTC](https://discuss.elastic.co/t/cross-cluster-search-switching-to-an-empty-backup-cluster/274601/2 "2021-06-02T02:28:25Z")

</div>

Welcome to our community! 😃

Can you share your CCR config?

---

<div class="post-metadata">

**Author:** ![driveirk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/driveirk/32/102253_2.png) [@driveirk](https://discuss.elastic.co/u/driveirk)\
**Post date:** [June 2, 2021, 3:33am UTC](https://discuss.elastic.co/t/cross-cluster-search-switching-to-an-empty-backup-cluster/274601/3 "2021-06-02T03:33:00Z")

</div>

I would not want to do a replication of each cluster. And I want to create one backup server for all clusters.  
The whole point of the task: when one cluster falls as a whole, Cross Cluster search does not work.  
Because there is no response from the fallen cluster.  
Therefore, for each cluster, an empty standby server with Elasticsearch is created in the haproxy config.  
But when switching to a backup server, the search also does not work and I get a 500 error.  
The error text is above.

The task of the backup server is to give an answer that there is no data.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [June 2, 2021, 3:36am UTC](https://discuss.elastic.co/t/cross-cluster-search-switching-to-an-empty-backup-cluster/274601/4 "2021-06-02T03:36:11Z")

</div>

Sorry, I meant CCS config.

---

<div class="post-metadata">

**Author:** ![driveirk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/driveirk/32/102253_2.png) [@driveirk](https://discuss.elastic.co/u/driveirk)\
**Post date:** [June 2, 2021, 4:39am UTC](https://discuss.elastic.co/t/cross-cluster-search-switching-to-an-empty-backup-cluster/274601/5 "2021-06-02T04:39:30Z")

</div>

```auto
cluster.name: test

node.name: ${HOSTNAME}
node.attr.host: ${HOSTNAME}
node.master: true
node.data: true
node.transform: false
node.remote_cluster_client: true
node.ml: false

path.data: /var/log/elasticsearch/data
path.logs: /var/log/elasticsearch

bootstrap.memory_lock: true

thread_pool.write.queue_size: 1000
indices.memory.index_buffer_size: 30%
indices.fielddata.cache.size: 40%
indices.queries.cache.count: 1000

network.host: 0.0.0.0
http.port: 9200
transport.tcp.port: 9300

discovery.seed_hosts: ["1.1.1.1", "2.2.2.2", "3.3.3.3"]
cluster.initial_master_nodes: ["${HOSTNAME}", "server1.ru", "server2.ru"]

#script.inline: true
#script.stored: true

xpack.security.enabled: false

async_search.index_cleanup_interval: "1m"

reindex.remote.whitelist: ["*:*"]
readonlyrest.settings_index: .readonlyrest

```

I've already tried disabling readonlyrest, it didn't help.

---

<div class="post-metadata">

**Author:** ![driveirk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/driveirk/32/102253_2.png) [@driveirk](https://discuss.elastic.co/u/driveirk)\
**Post date:** [June 2, 2021, 4:45am UTC](https://discuss.elastic.co/t/cross-cluster-search-switching-to-an-empty-backup-cluster/274601/6 "2021-06-02T04:45:39Z")

</div>

Here is the connection log:

```auto

[2021-06-02T04:41:21,683][WARN][r.suppressed] [test]path: /cluster1:*log*/_search, params: {pretty=, size=1, index=cluster1:*log*}
org.elasticsearch.transport.RemoteTransportException: [error while communicating with remote cluster [cluster1]]
Caused by: java.lang.IllegalStateException: Unable to open any proxy connections to remote cluster [cluster1]
	at org.elasticsearch.transport.ProxyConnectionStrategy.openConnections(ProxyConnectionStrategy.java:244) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.transport.ProxyConnectionStrategy.access$000(ProxyConnectionStrategy.java:51) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.transport.ProxyConnectionStrategy$1.onFailure(ProxyConnectionStrategy.java:210) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.transport.ProxyConnectionStrategy$2.onFailure(ProxyConnectionStrategy.java:237) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:88) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98) [elasticsearch-7.8.1.jar:7.8.1]
	at java.util.ArrayList.forEach(ArrayList.java:1510) [?:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.common.util.concurrent.BaseFuture.setException(BaseFuture.java:162) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onFailure(ListenableFuture.java:135) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.transport.ClusterConnectionManager.failConnectionListeners(ClusterConnectionManager.java:273) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.transport.ClusterConnectionManager.lambda$connectToNode$2(ClusterConnectionManager.java:169) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:71) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.action.ActionListener$4.onFailure(ActionListener.java:173) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.action.ActionListener$4.onResponse(ActionListener.java:159) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.transport.TransportService$5.onResponse(TransportService.java:475) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.transport.TransportService$5.onResponse(TransportService.java:465) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:54) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1163) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:213) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:695) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.8.1.jar:7.8.1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
	at java.lang.Thread.run(Thread.java:832) [?:?]

```

Let me remind you that restarting ElasticSearch through which the search is done solves the problem.  
But when the cluster goes up, the error is returned.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [June 2, 2021, 4:47am UTC](https://discuss.elastic.co/t/cross-cluster-search-switching-to-an-empty-backup-cluster/274601/7 "2021-06-02T04:47:57Z")

</div>

That's not your cross cluster search config, we need to see that please.

---

<div class="post-metadata">

**Author:** ![driveirk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/driveirk/32/102253_2.png) [@driveirk](https://discuss.elastic.co/u/driveirk)\
**Post date:** [June 2, 2021, 4:59am UTC](https://discuss.elastic.co/t/cross-cluster-search-switching-to-an-empty-backup-cluster/274601/8 "2021-06-02T04:59:37Z")

</div>

Thank you for giving me a thought  
skip\_unavailable solved my problem.

```auto
        "lab" : {
          "mode" : "proxy",
          "skip_unavailable" : "true",
          "proxy_address" : "127.0.0.1:9101"
        },

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 30, 2021, 5:00am UTC](https://discuss.elastic.co/t/cross-cluster-search-switching-to-an-empty-backup-cluster/274601/9 "2021-06-30T05:00:05Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.