Hello, I have 2 K8s clusters, Cluster A and Cluster B. I deployed Metricbeat on each cluster by using this yaml file: beats/deploy/kubernetes/metricbeat-kubernetes.yaml at v8.12.2 · elastic/beats · GitHub
For metricbeat.yml file, I added the following extra configuration:
processors:
- add_tags:
tags: [cluster-dev]
- add_cloud_metadata:
output.elasticsearch:
index: "cluster-dev-metrics-%{[agent.version]}"
hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}']
username: ${ELASTICSEARCH_USERNAME}
password: ${ELASTICSEARCH_PASSWORD}
setup:
kibana.host: "https://kibana.example.com:443"
dashboards.index: "cluster-dev-metrics-*"
dashboards.enabled: true
template.name: "cluster-dev-metrics-%{[agent.version]}"
template.pattern: "cluster-dev-metrics-%{[agent.version]}"
This was done on each cluster, and can I see the kube-state-metrics and metricbeat pods as healthy. On Kibana, I see the 2 data views that are created, then I go to Discover and I see the metrics from Cluster A and the metrics from Cluster B. Since I added a different tag on each metricbeat.yml file, I can confirm that metrics from Cluster A have the correct tag, the same is true for Cluster B.
At this point, I have the metrics from the 2 clusters, that go to the Elasticsearch instance and I see them in Kibana Discover tab.
Now if I go to Kibana Dashboards and click on the newly created "[Metricbeat Kubernetes] Overview" dashboard, I see the data only from the cluster, on which I deployed metribeat most recently. And this is my problem. If I deploy metricbeat on Cluster A, I see that data from Cluster A in the dashboard. If I then deploy metricbeat on Cluster B, I only see data from Cluster B in the dashboard. I am not able to see data from both clusters in the Kibana Kubernetes dashboard.
This appears to be an intended functionality, because on this doc page Configure Kibana dashboard loading | Metricbeat Reference [8.12] | Elastic it says:
When dashboard loading is enabled, Metricbeat overwrites any existing dashboards that match the names of the dashboards you are loading. This happens every time Metricbeat starts.
And since I have dashboards.enabled: true on both metricbeat.yml files, the latest deployment of metricbeat, overwrites the previous dashboards in Kibana.
What I want, is to configure the 2 instances of metricbeat, so that data from both Cluster A and Cluster B can be displayed in the "[Metricbeat Kubernetes] Overview" dashboard. I know that this should be possible, since this dashboard allows the user to select a cluster from a dropdown box list as you can see in this picture:
Do you have any suggestions on how to get the data from both clusters and be able to display them in the "[Metricbeat Kubernetes] Overview" dashboard, by choosing the corresponding cluster by its Cluster Name?
On this page Metricbeat setup overwrite Elastic Agent built in UI (Dashboard and Visualization) someone mentioned:
As a workaround you can try to setup the agent again and modifying the Metricbeat dashboard id of the dashboard that is being overridden. The dashboard file info is in kibana/7/dashboard folder of Metricbeat.
Would it be possible to configure the metricbeat.yml file, so that the dashboard IDs are different? That may be a valid workaround because I could have 2 links for 2 dashboards in Kibana, each one getting data from the corresponding cluster.
Thank you.