Kibana monitoring - (wrong) data from remote cluster

Elastic Stack Kibana
1

Hi.
We have this setup:

  • cluster 1 (cluster_name: reglendo-es-cluster), 4 nodes
  • cluster 2 (cluster_name: halda-cluster), 1 node
  • kibana 1 connected to cluster 1
  • kibana 2 connected to cluster 2
  • setup cross-cluster search from cluster 2 to cluster 1 (cluster 2 is for "cold" data and cluster 1 for production data - we move the data manually from production to cold)
  • ports 9200+9300 from cluster 1 are accessible from cluster 2 (and kibana 2 - its same server)

The problem is, that when I open kibana 2, I can see documents from cluster 2, dashboards,... but when click to Monitoring, I see data from cluster 1 and cant switch to cluster 2. I have no idea how to switch this... (btw in kibana 1 is nothing wrong.. everything works as expected).

Have you any idea how to solve this issue?

Edit:
I also look into indices .monitoring-* on cluster 1 and there is really only cluster_uuid of cluster 1. But on cluster 2, there are no indices "monitoring-*"

2

Hi @kluvi,

A few follow up questions to try and help clarify the picture:

  1. Which versions of ES and Kibana are you running?

  2. Can you provide a screenshot of the Monitoring UI on Kibana 2?

  3. Can you provide your kibana.yml for kibana 1 and kibana 2, after masking any sensitive information?

  4. For each cluster, can your provide the results of this query: GET _cluster/settings?include_defaults&filter_path=*.xpack.monitoring.*

3

cluster 2 is 6.5.4
cluster 1 is 6.4.1 (we will update it today also to 6.5.4)

image
image947×713 56.5 KB

(URL is https://XXXXXXX/app/monitoring#/overview?_g=(ccs:reglendo-es-cluster,cluster_uuid:'8EpTnISFS8qg03qU1YxWUw') where XXXXXXX is domain of kibana 2)

We run Kibana in docker container (image docker.elastic.co/kibana/kibana:6.5.4 without any custom modifications) and set only environment variables SERVER_NAME and ELASTICSEARCH_URL

cluster 1:

image
image383×794 33 KB

cluster 2:
image
image369×945 39.2 KB

I see, that on cluster 2, there is xpack.monitoring.collection.enabled: false, but its weird, because its only in defaults section... Why is it different on each cluster?

btw I updated the settings through PUT /_cluster/settings and ES creates the .monitoring-* index and now I can see correct cluster in Monitoring in Kibana.

So thank you for your hints