Cross Cluster search, switching to an empty backup cluster

Elastic Stack Elasticsearch
1

Hello,
I want to make a backup cluster for Cross Cluster search.
I selected proxy_mod, I use Haproxy as a proxy

Haproxy configuration example:

server work 1.1.1.1:9300 check on-marked-up shutdown-backup-sessions on-marked-down shutdown-sessions
server backup 2.2.2.2:9300 check backup on-marked-up shutdown-backup-sessions on-marked-down shutdown-sessions

When cluster "work" is unavailable, it switches to the backup server,
But with this in elastic, I get the error:

    {
       "error": {
         "root_cause": [
           {
             "type": "illegal_state_exception",
             "reason": "Unable to open any proxy connections to remote cluster [lab]"
           }
         ],
         "type": "illegal_state_exception",
         "reason": "Unable to open any proxy connections to remote cluster [lab]"
       },
       "status": 500
    }

Restarting elasticity solves the problem, but if the traffic is switched back, the problem is reproduced again.

Importantly, the "work" and "backup" clusters are not related in any way.

2

Welcome to our community! :smiley:

Can you share your CCR config?

3

I would not want to do a replication of each cluster. And I want to create one backup server for all clusters.
The whole point of the task: when one cluster falls as a whole, Cross Cluster search does not work.
Because there is no response from the fallen cluster.
Therefore, for each cluster, an empty standby server with Elasticsearch is created in the haproxy config.
But when switching to a backup server, the search also does not work and I get a 500 error.
The error text is above.

The task of the backup server is to give an answer that there is no data.

4

Sorry, I meant CCS config.

5 
cluster.name: test

node.name: ${HOSTNAME}
node.attr.host: ${HOSTNAME}
node.master: true
node.data: true
node.transform: false
node.remote_cluster_client: true
node.ml: false

path.data: /var/log/elasticsearch/data
path.logs: /var/log/elasticsearch

bootstrap.memory_lock: true

thread_pool.write.queue_size: 1000
indices.memory.index_buffer_size: 30%
indices.fielddata.cache.size: 40%
indices.queries.cache.count: 1000

network.host: 0.0.0.0
http.port: 9200
transport.tcp.port: 9300

discovery.seed_hosts: ["1.1.1.1", "2.2.2.2", "3.3.3.3"]
cluster.initial_master_nodes: ["${HOSTNAME}", "server1.ru", "server2.ru"]

#script.inline: true
#script.stored: true

xpack.security.enabled: false

async_search.index_cleanup_interval: "1m"

reindex.remote.whitelist: ["*:*"]
readonlyrest.settings_index: .readonlyrest

I've already tried disabling readonlyrest, it didn't help.

6

Here is the connection log:


[2021-06-02T04:41:21,683][WARN ][r.suppressed             ] [test]path: /cluster1:*log*/_search, params: {pretty=, size=1, index=cluster1:*log*}
org.elasticsearch.transport.RemoteTransportException: [error while communicating with remote cluster [cluster1]]
Caused by: java.lang.IllegalStateException: Unable to open any proxy connections to remote cluster [cluster1]
	at org.elasticsearch.transport.ProxyConnectionStrategy.openConnections(ProxyConnectionStrategy.java:244) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.transport.ProxyConnectionStrategy.access$000(ProxyConnectionStrategy.java:51) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.transport.ProxyConnectionStrategy$1.onFailure(ProxyConnectionStrategy.java:210) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.transport.ProxyConnectionStrategy$2.onFailure(ProxyConnectionStrategy.java:237) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.action.ActionRunnable.onFailure(ActionRunnable.java:88) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:39) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:226) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98) [elasticsearch-7.8.1.jar:7.8.1]
	at java.util.ArrayList.forEach(ArrayList.java:1510) [?:?]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.common.util.concurrent.BaseFuture.setException(BaseFuture.java:162) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.common.util.concurrent.ListenableFuture.onFailure(ListenableFuture.java:135) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.transport.ClusterConnectionManager.failConnectionListeners(ClusterConnectionManager.java:273) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.transport.ClusterConnectionManager.lambda$connectToNode$2(ClusterConnectionManager.java:169) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.action.ActionListener$1.onFailure(ActionListener.java:71) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.action.ActionListener$4.onFailure(ActionListener.java:173) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.action.ActionListener$4.onResponse(ActionListener.java:159) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.transport.TransportService$5.onResponse(TransportService.java:475) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.transport.TransportService$5.onResponse(TransportService.java:465) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:54) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1163) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.transport.InboundHandler$1.doRun(InboundHandler.java:213) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:695) [elasticsearch-7.8.1.jar:7.8.1]
	at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.8.1.jar:7.8.1]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
	at java.lang.Thread.run(Thread.java:832) [?:?]

Let me remind you that restarting ElasticSearch through which the search is done solves the problem.
But when the cluster goes up, the error is returned.

7

That's not your cross cluster search config, we need to see that please.

1 Like
8

Thank you for giving me a thought
skip_unavailable solved my problem.

        "lab" : {
          "mode" : "proxy",
          "skip_unavailable" : "true",
          "proxy_address" : "127.0.0.1:9101"
        },
9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.