I am receiving the following error on the Monitoring page in Kibana after configuring TLS to encrypt communications between nodes in the cluster.
Monitoring Request Error
[transport_exception] handshake failed because connection reset: Check the Elasticsearch Monitoring cluster network connection or the load level of the nodes.
HTTP 503
Background:
I set up a new elasticsearch 6.8.8 cluster with a separate server running Kibana. I have 3 master nodes, 2 data nodes, and 2 coordinating nodes plus the server running Kibana.
In order to apply my license, I had to set up TLS and followed the documentation to configure. I applied the configurations to all of the elasticsearch nodes in the cluster, and only the elasticsearch nodes.
I thought perhaps I had to apply the xpack transport configurations to Kibana so I did but this caused further issues (i.e. Kibana would not start). After reviewing this other discussion thread I see that these settings aren't supported by Kibana and I removed them (i.e. I removed the xpack.security.transport.ssl.* lines from kibana.yml and removed the xpack keystores I created).
I'm now back where I started. Kibana starts, I can sign in, but the Monitoring page returns an HTTP 503.
I have confirmed that the cluster is healthy:
$ curl -X GET -k "https://master-0:9200/_cat/nodes?pretty"
10.10.1.5 30 86 1 0.07 0.05 0.04 di - data-0
10.10.1.10 5 95 1 0.13 0.05 0.01 i - client-1
10.10.1.7 5 95 0 0.13 0.03 0.01 mi - master-1
10.10.1.8 4 95 0 0.03 0.01 0.00 mi - master-2
10.10.1.9 5 95 1 0.00 0.02 0.00 mi * master-0
10.10.1.11 5 95 0 0.01 0.06 0.02 i - client-0
10.10.1.6 29 85 0 0.11 0.04 0.01 di - data-1
$ curl -X GET -k "https://master-0:9200/_cat/health?pretty"
1587153023 19:50:23 elasticsearch green 7 2 30 15 0 0 0 0 - 100.0%
What is the Elasticsearch Monitoring cluster network connection? How can I resolve this issue?