I'm experimenting with the cross-cluster search feature and I have a question about error handling.
Imagine I have a multi-cluster setup like this:
- cluster_master: my app connects here and this is where I setup the connections to the other clusters
- cluster_two: some data, say from customer A
- cluster_three: some data, say from customer B
I need to be able to query cluster_master regardless of the status of the connection to cluster_two and cluster_three. In other words, it's not acceptable, in my situation, that the whole search breaks if cluster_three cannot be reached for 10 seconds, 10 minutes or 10 days. The "whole" need to keep working and I would just get less data, possibly with some indication that not all clusters were reacheable.
What is the best way to accomplish this, beside trying to adjust the query dynamically after parsing the error texts that I get back (i.e. "unable to communicate with remote cluster [cluster_XXX]" )?