Cross-Cluster search Implementation Issues

[tusharnemade]

Hello Team

We have implemented Cross-Cluster Search in Elasticsearch version 7.8.0 clusters.

For example Cluster-A and Cluster-B.

Cluster-A elasticsearch.yml file has below parameters :

cluster.remote.Cluster-B.seeds: 10.10.10.20:9300
cluster.remote.Cluster-B.transport.ping_schedule: 60s

Cluster-B Does not have parameters for Cluster-A , as its only one-side cross-cluster search implemented.

Now Cluster-A and Cluster-B were down for some maintenance.

Started Cluster-A , first , it gave below error :

[2021-04-20T03:33:25,375][WARN ][o.e.t.SniffConnectionStrategy] [node-1] fetching nodes from external cluster [cls_pp_applog] failed
org.elasticsearch.transport.ConnectTransportException: [][10.10.10.20:9300] connect_exception
...
...
Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: 10.10.10.20:9300
Caused by: java.net.ConnectException: Connection refused

[2021-04-20T03:33:25,414][INFO ][o.e.n.Node               ] [node-1] stopping ...
[2021-04-20T03:33:25,420][INFO ][o.e.n.Node               ] [node-1] stopped
[2021-04-20T03:33:25,420][INFO ][o.e.n.Node               ] [node-1] closing ...
[2021-04-20T03:33:25,465][INFO ][o.e.n.Node               ] [node-1] closed

I want to confirm , in Cross-Cluster Search Implementation , its mandatory to have CLUSTER-B should be up and running , otherwise Cluster-A will be shutdown ?

[warkolm (Mark Walkom)]

Elasticsearch won't stop itself like that, something has asked the process to stop.
You might want to check your OS logs to see if there is anything related.

[tusharnemade]

OS Logs are clear.

Nothing found.

This is happened when i started Node-1 of Cluster-A with

elasticsearch -d command....

During this time Cluster-B was completely down.

[tusharnemade]

@tusharnemade

Caused by: io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: 10.10.10.20:9300
Caused by: java.net.ConnectException: Connection refused

Above Error is the only indicator of why Elasticsearch daemon on this node went down.

Its Nothing to do with any other factor or OS.

[DavidTurner]

@tusharnemade

in Cross-Cluster Search Implementation , its mandatory to have CLUSTER-B should be up and running , otherwise Cluster-A will be shutdown ?

Only if you define the remote cluster in elasticsearch.yml. If you define it via PUT _cluster/settings then no. See this issue for more details:

Cross Cluster Search: remote clusters need to be available when registered in config file only

[tusharnemade]

@DavidTurner

_cluster/settings

So we have limitation in here. We want things to be on elasticsearch.yml so that its available to view settings of cluster at one place.

Lastly, Persistent settings will do or only transient ?

curl -X PUT "localhost:9200/_cluster/settings?pretty" -H 'Content-Type: application/json' -d'
{
  "persistent" : {
    "cluster.remote.Cluster-B.seeds" : "10.10.10.20:9300" ,
    "cluster.remote.Cluster-B.transport.ping_schedule" : "60s"
  }
}
'

[DavidTurner]

Either is fine, it depends whether you want them to persist across restarts or not. Usually you do.

[tusharnemade]

Okay.

Thank You @DavidTurner for your responses and help.