Kibana Cannot open monitoring dashboard because remote cluster is not accessable

[edwardcccc (Chen Chen)]

I have a ELK 8.4.3 Main Cluster
I have ELK self-monitoring enabled on this cluster

another ELK 8.4.1 Remote Cluster, on a separate machine, which is not online all the time
I only use this cluster to offload data from main cluster

---

The problem only occurs when the remote cluster is offline.

When I try to access the monitoring dashboard on the Main Cluster, I got nothing and an error.
The error logs are shown in the screenshot

I think remote cluster does not have anything to do with self-monitoring, but this is preventing me from using it

[swchandu (chandrakanth)]

I feel that there is some issue with name resolving/connectivity/route tables.
Because it is giving "No route to host".
Try to check whether that IP(x.x.x.66) port 9300 is connecting.

[edwardcccc (Chen Chen)]

The problem only occurs when the remote cluster is offline.

[matschaffer]

Hi @edwardcccc ,

Is the Remote set with skip_unavailable: true?

If it is and you're still seeing failures you may want to set the tcp retries on the Main cluster OS as shown on TCP retransmission timeout | Elasticsearch Guide [8.4] | Elastic

As long as the Main cluster can respond to a query when the Remote cluster is down, the stack monitoring UI should load.

Alternatively if the .monitoring-* indices all stay on the Main cluster, you can set monitoring.ui.ccs.enabled: false (reference) which will make Stack Monitoring UI load using only indices from the Main cluster.