Cluster#1 7.10.0
Kibana#1 7.10.0
Cluster#2 7.9.2
Kibana#2 7.9.2
Problem#1
I have a local Kibana (kibana#1) to a local cluster (cluster#1) and everything works perfectly on querying an index pattern. I set up a different cluster (cluster#2) and an instance of Kibana (kibana#2) not on the same network, added the remote cluster (cluster#1) to the kibana (kibana#2). Create an index pattern for an index on (cluster#1) every query comes back with shards failed and a null_pointer_exception error. Neither the remote ES cluster#1 or the local cluster#2 to the remote kibana#2 throws any errors in their respective cluster logs or gc logs. Kibana#2 standard out throws no errors either as well as kibana.stderr. I have stopped Kibana#2, deleted every ".kibana*" indice on the cluster#2, restarted kibana#2, reset up the remote index pattern to cluster#1and same thing, shards fail with null_pointer_exception. Cluster#1 is healthy with zero realocating, initializing, unassigned and delayed_unassigned shards. The same for Cluster #2.
Additionally after the index pattern is created, I killed my browser session with kibana#2 and setup tcpdump to watch traffic flows on port 9300 between cluster#1 & cluster#2. Relaunch my browser connection to kibana#2 and I get the same failure with NO traffic flow on 9300 between the two clusters.
Problem#2
I upgraded Cluster#2 & Kibana#2 to see if would fix the issue and it created another issue, cannot create any new remote cluster index patterns on kibana#2 to cluster#1 because system has unending "Looking for matching indices". I had this problem before problem#1 and the only thing that fixed with was starting over with a backup of another vm that works and changing the IPs and deleting all indices, but after completion I encountered problem#1.
Help....