# Metricbeat dashboard strange behaviour **URL:** \ **Category:** Beats\ **Created:** [March 16, 2021, 4:42pm UTC](https://discuss.elastic.co/t/metricbeat-dashboard-strange-behaviour/267401 "2021-03-16T16:42:39Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![alfredo.deluca](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alfredo.deluca/32/95381_2.png) [@alfredo.deluca](https://discuss.elastic.co/u/alfredo.deluca)\ **Post date:** [March 16, 2021, 4:42pm UTC](https://discuss.elastic.co/t/metricbeat-dashboard-strange-behaviour/267401/1 "2021-03-16T16:42:39Z")
Hi all. I am using [elastic.co](http://elastic.co) cloud 7.11.1 and I have 2 different metricbeat sitting in 2 different k8s clusters sending all the metrics to ELK. The first one works fine and I can see all the metrics in kibana but when I turn the second metricbeat all the node (but also all the other metrics ) will be added to the dashboard which is wrong. Both metricebeats have different index. What is wrong? For instance, if I have 6 k8s nodes in one cluster and 9 in the 2nd one, in the dashboard I ll see 15 nodes. The first metrics is on one index metricbeat-7.10.2-2021.01.19-000001 and the second one is essentia-metrics-dev-7.10.2-2021.03.16. any clue? Cheers ---
**Author:** ![alfredo.deluca](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alfredo.deluca/32/95381_2.png) [@alfredo.deluca](https://discuss.elastic.co/u/alfredo.deluca)\ **Post date:** [March 17, 2021, 10:27am UTC](https://discuss.elastic.co/t/metricbeat-dashboard-strange-behaviour/267401/2 "2021-03-17T10:27:21Z")
Hi all. anyone on this? ---
**Author:** ![alfredo.deluca](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alfredo.deluca/32/95381_2.png) [@alfredo.deluca](https://discuss.elastic.co/u/alfredo.deluca)\ **Post date:** [March 22, 2021, 10:24pm UTC](https://discuss.elastic.co/t/metricbeat-dashboard-strange-behaviour/267401/3 "2021-03-22T22:24:30Z")
what a strange thing. Even if I put tags, different index and different space id ...still I get ALL the metrics from different k8s cluster in the same dashboard. anyone can explain? cheers ---
**Author:** ![eedugon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eedugon/32/45404_2.png) [@eedugon](https://discuss.elastic.co/u/eedugon)\ **Post date:** [April 14, 2021, 12:06pm UTC](https://discuss.elastic.co/t/metricbeat-dashboard-strange-behaviour/267401/4 "2021-04-14T12:06:58Z")
@alfredo.deluca , next week I'd like to reproduce your setup in a lab. Would you please share the configuration of the 2 different metricbeats? Also include some screenshots and details about the dashboards, to understand what exact dashboard you are using (note: shouldn't you create 2 different dashboards / visualizations to access specific clusters data?). When fetching data (metrics & logs) from multiple kubernetes clusters at the moment there isn't any easy way to differentiate them, as in the schema there isn't anything like a cluster\_id or cluster\_name at k8s level. But if you are sending the data to different indices, and if the dashboards and visualizations are configured to access the right index pattern I agree you should be able to control what cluster to visualize. Also please confirm that you are using a standard dashboards and not the `Metrics UI` inside the `Observability App`. ---
**Author:** ![alfredo.deluca](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alfredo.deluca/32/95381_2.png) [@alfredo.deluca](https://discuss.elastic.co/u/alfredo.deluca)\ **Post date:** [April 14, 2021, 1:17pm UTC](https://discuss.elastic.co/t/metricbeat-dashboard-strange-behaviour/267401/5 "2021-04-14T13:17:54Z")
Thanks Eduardo. I will share my conf and more details later on. For sure we are using different indices for 2 different metricbeat/dashboards and I am not using the Observability App. Cheers Cheers ---
**Author:** ![alfredo.deluca](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alfredo.deluca/32/95381_2.png) [@alfredo.deluca](https://discuss.elastic.co/u/alfredo.deluca)\ **Post date:** [April 14, 2021, 4:51pm UTC](https://discuss.elastic.co/t/metricbeat-dashboard-strange-behaviour/267401/6 "2021-04-14T16:51:58Z")
Hi Eduardo. So I use metricbeat helm 7.10.2 (I will upgrade to 7.12 soonish) and basically my 2 values.yaml from 2 different kubernetes clusters are in attachment. They are pretty much identical except the index,index\_patttern etc etc are different (you'll see both in the daemonset and deployment section). Of course I omitted the [cloud.id](http://cloud.id) and cloud.auth. As far as I understand, when metricbeat starts, uploads all the default dashboards and visualize libraries including, of course, those for Kubernetes. Let me know if you have any other questions. Cheers (Attachment essentia.yaml is missing) (Attachment nlapi.yaml is missing) ---
**Author:** ![alfredo.deluca](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alfredo.deluca/32/95381_2.png) [@alfredo.deluca](https://discuss.elastic.co/u/alfredo.deluca)\ **Post date:** [April 14, 2021, 7:22pm UTC](https://discuss.elastic.co/t/metricbeat-dashboard-strange-behaviour/267401/7 "2021-04-14T19:22:50Z")
sorry.. got blocked the yaml files. here are the pastebin ref. > **[\---daemonset: # Annotations to apply to the daemonset annotations: {} -...](https://pastebin.com/MV49491Z)** > > Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time. > **[\---daemonset: # Annotations to apply to the daemonset annotations: {} -...](https://pastebin.com/xfbXDMRX)** > > Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time. ---
**Author:** ![eedugon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eedugon/32/45404_2.png) [@eedugon](https://discuss.elastic.co/u/eedugon)\ **Post date:** [April 19, 2021, 9:10am UTC](https://discuss.elastic.co/t/metricbeat-dashboard-strange-behaviour/267401/8 "2021-04-19T09:10:45Z")
Thanks for sharing the manifests @alfredo.deluca ! However I still need to see the issue, as I'd like to check what type of dashboards we are talking about and how exactly the visualizations are configured (the `index-patterns` being pointed by the visualizations). Would you please share some screenshots showing the problem? - Show a screenshot showing the 15 hosts in a dashboard. - Then Edit the visualization and show the associated index pattern. FYI, I have reproduced your setup using a similar configuration and all worked great with `7.11.2`. Here you have my setup: - Kibana: Already created spaces `cluster1` and `cluster2`. - Cluster1 Metricbeat: ```auto setup.ilm.enabled: false setup.kibana: host: "https://MY-CLOUD-KIBANA:9243" space.id: "cluster1" setup.template.name: "cluster1" setup.template.pattern: "cluster1-metricbeat-dev-*" setup.dashboards: enabled: true beat: metricbeat index: "cluster1-metricbeat-dev-*" output.elasticsearch: hosts: ["https://MY-CLOUD-CLUSTER:9243"] index: "cluster1-metricbeat-dev-%{[agent.version]}-%{+yyyy.MM.dd}" username: "elastic" password: "my_password" ``` - Cluster2 Metricbeat: ```auto setup.ilm.enabled: false setup.kibana: host: "https://MY-CLOUD-KIBANA:9243" space.id: "cluster2" setup.template.name: "cluster2" setup.template.pattern: "cluster2-metricbeat-dev-*" setup.dashboards: enabled: true beat: metricbeat index: "cluster2-metricbeat-dev-*" output.elasticsearch: hosts: ["https://MY-CLOUD-CLUSTER:9243"] index: "cluster2-metricbeat-dev-%{[agent.version]}-%{+yyyy.MM.dd}" username: "elastic" password: "my_password" ``` I built this based on your config files (btw, nice plan about separating the dashboards in different spaces) and the dashboards + index patterns were created in the right dashboards. Also each of the dashboards was pointing correctly to the right pattern. Attaching some screenshots of the result, showing `cluster1` space:: - Data split in multiple indices: - Cluster1 `Nodes` Visualization of default `Kubernetes Overview ECS` dashboard. We can see how it points to the right pattern: ![eduardo](https://us1.discourse-cdn.com/elastic/original/3X/f/5/f57a0d35ad28ee40c2861eb759fc0c9835e1f7ca.png) - Cluster1 Dashboard: ![SUPPORTLAB](https://us1.discourse-cdn.com/elastic/original/3X/1/c/1c2851ccd1a6cc33f1734ae50ce3c918dd99c596.png) Cluster2 space looks as expected, using its own pattern: ![Panel options](https://us1.discourse-cdn.com/elastic/original/3X/4/0/4090e4cf0ef20a657769859f5ccbff3c3acbfdcf.png) As a final comment, in the default namespace all the dashboards were created pointing to the default data `metricbeat-*` (because of my initial config for testing purposes), which obviously didn't show any data. I updated manually some visualizations to point to `cluster*` and that made to obtain the data from all clusters together, also as expected. I hope this helps you to find out why exactly your dashboards are showing the combined data. Once we clarify that we should be able to determine the best movement. Please share some details of your issue from the dashboards and visualizations point of view if you want me to take a deeper look. ---
**Author:** ![alfredo.deluca](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alfredo.deluca/32/95381_2.png) [@alfredo.deluca](https://discuss.elastic.co/u/alfredo.deluca)\ **Post date:** [April 20, 2021, 11:01am UTC](https://discuss.elastic.co/t/metricbeat-dashboard-strange-behaviour/267401/9 "2021-04-20T11:01:11Z")
Hi @eedugon thanks for your info. I will go through your email later on. For now - when you set **beat: metricbeat** is just to upload the metricbeat dashboars and not all the others? - Every time you start the metricbeat, will it upload and override the existing dashboards as fas as you know? Cheers ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 18, 2021, 11:01am UTC](https://discuss.elastic.co/t/metricbeat-dashboard-strange-behaviour/267401/10 "2021-05-18T11:01:58Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.