Kibana monitoring - (wrong) data from remote cluster

Hi.
We have this setup:

  • cluster 1 (cluster_name: reglendo-es-cluster), 4 nodes
  • cluster 2 (cluster_name: halda-cluster), 1 node
  • kibana 1 connected to cluster 1
  • kibana 2 connected to cluster 2
  • setup cross-cluster search from cluster 2 to cluster 1 (cluster 2 is for "cold" data and cluster 1 for production data - we move the data manually from production to cold)
  • ports 9200+9300 from cluster 1 are accessible from cluster 2 (and kibana 2 - its same server)

The problem is, that when I open kibana 2, I can see documents from cluster 2, dashboards,... but when click to Monitoring, I see data from cluster 1 and cant switch to cluster 2. I have no idea how to switch this... (btw in kibana 1 is nothing wrong.. everything works as expected).

Have you any idea how to solve this issue?

Edit:
I also look into indices .monitoring-* on cluster 1 and there is really only cluster_uuid of cluster 1. But on cluster 2, there are no indices "monitoring-*"

Hi @kluvi,

A few follow up questions to try and help clarify the picture:

  1. Which versions of ES and Kibana are you running?

  2. Can you provide a screenshot of the Monitoring UI on Kibana 2?

  3. Can you provide your kibana.yml for kibana 1 and kibana 2, after masking any sensitive information?

  4. For each cluster, can your provide the results of this query: GET _cluster/settings?include_defaults&filter_path=*.xpack.monitoring.*

cluster 2 is 6.5.4
cluster 1 is 6.4.1 (we will update it today also to 6.5.4)


(URL is https://XXXXXXX/app/monitoring#/overview?_g=(ccs:reglendo-es-cluster,cluster_uuid:'8EpTnISFS8qg03qU1YxWUw') where XXXXXXX is domain of kibana 2)

We run Kibana in docker container (image docker.elastic.co/kibana/kibana:6.5.4 without any custom modifications) and set only environment variables SERVER_NAME and ELASTICSEARCH_URL

cluster 1:


cluster 2:

I see, that on cluster 2, there is xpack.monitoring.collection.enabled: false, but its weird, because its only in defaults section... Why is it different on each cluster?

btw I updated the settings through PUT /_cluster/settings and ES creates the .monitoring-* index and now I can see correct cluster in Monitoring in Kibana.

So thank you for your hints

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.