# Kubernetes pods found, but all metrics 0% **URL:** \ **Category:** Metrics\ **Created:** [May 7, 2019, 8:46pm UTC](https://discuss.elastic.co/t/kubernetes-pods-found-but-all-metrics-0/180061 "2019-05-07T20:46:47Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![willemdh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/willemdh/32/16922_2.png) [@willemdh](https://discuss.elastic.co/u/willemdh)\ **Post date:** [May 7, 2019, 8:46pm UTC](https://discuss.elastic.co/t/kubernetes-pods-found-but-all-metrics-0/180061/1 "2019-05-07T20:46:47Z")
Hello, Did some tests on the new Infrastructure ui in Kibana 6.7.1. Hosts metrics are working fine, but Kubernetes metrics seem to be missing some data. Although the pod names, nodes and namespaces are recognised, all metrics report 0%, both for cpu, memory, the inbound / outbound traffic all show 0bit/s. Metricbeat data is generated on 1 master Openshift node with following module config: ``` - module: kubernetes metricsets: - node - system - pod - container - volume period: 10s in_cluster: false add_metadata: true kube_config: ../.kube/config host: "node1" hosts: ["https://node1:10250","https://node2:10250",.......] ssl.certificate_authorities: ["/etc/pki/ca-trust/source/anchors/openshift-ca.crt"] ssl.certificate: "crt" ssl.key: "key" ``` ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/1/61ea10163618ae33886e8bf3eb1a46294eb75f3c.png) All kubernetes metricbeat data is in metricbeat-\*. CHecking the data in discovery, I can find related metrics: ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/9/59af59e5993b3f2974d7db530f6ffb342739e0f0.png) As you can see, we are not running a daemonset (as this caused us too many headaches in Openshift), could it be related to `in_cluster: false`? Is there something else which can cause the kubernetes metrics in the Infrastructure ui to be '0'? Thanks. Willem ---
**Author:** ![simianhacker](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/simianhacker/32/3383_2.png) [@simianhacker](https://discuss.elastic.co/u/simianhacker)\ **Post date:** [May 7, 2019, 9:31pm UTC](https://discuss.elastic.co/t/kubernetes-pods-found-but-all-metrics-0/180061/2 "2019-05-07T21:31:57Z")
@willemdh When you click on the metric detail pages, are the charts empty? ---
**Author:** ![willemdh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/willemdh/32/16922_2.png) [@willemdh](https://discuss.elastic.co/u/willemdh)\ **Post date:** [May 8, 2019, 6:41am UTC](https://discuss.elastic.co/t/kubernetes-pods-found-but-all-metrics-0/180061/3 "2019-05-08T06:41:34Z")
@simianhacker Thanks for the fast reply. When I click on the metric details page, There seems to be no data.. ``` ## There is no data to display. Try adjusting your time or filter. ``` Please suggest what else I should check to get this working. Best regards, Willem ---
**Author:** ![willemdh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/willemdh/32/16922_2.png) [@willemdh](https://discuss.elastic.co/u/willemdh)\ **Post date:** [May 8, 2019, 8:43am UTC](https://discuss.elastic.co/t/kubernetes-pods-found-but-all-metrics-0/180061/4 "2019-05-08T08:43:22Z")
@simianhacker Found the cause of the problem. We only run the Metricbeat kubernetes module on 1 master node with: ``` - module: kubernetes metricsets: - node - system - pod - container - volume period: 10s in_cluster: false add_metadata: true kube_config: ../.kube/config host: "node1" hosts: ["https://node1:10250","https://node2:10250",.......] ssl.certificate_authorities: ["/etc/pki/ca-trust/source/anchors/openshift-ca.crt"] ssl.certificate: "crt" ssl.key: "key" ``` So it seem like: `hosts: ["https://node1:10250","https://node2:10250",.......]` work to get the metrics from the pods, but not to get the metadata for the metrics of those pods. The metadata is only added for the host in: `host: "node1"` I was able to workaround this issue by duplicating the kubernetes module config for each Openshift node like this: ``` - module: kubernetes metricsets: - node - system - pod - container - volume period: 10s in_cluster: false add_metadata: true kube_config: ../.kube/config host: "node1" hosts: ["https://node1:10250"] ssl.certificate_authorities: ["/etc/pki/ca-trust/source/anchors/openshift-ca.crt"] ssl.certificate: "crt" ssl.key: "key" - module: kubernetes metricsets: - node - system - pod - container - volume period: 10s in_cluster: false add_metadata: true kube_config: ../.kube/config host: "node2" hosts: ["https://node2:10250"] ssl.certificate_authorities: ["/etc/pki/ca-trust/source/anchors/openshift-ca.crt"] ssl.certificate: "crt" ssl.key: "key" ``` This seem weird.. Why would Elastic allow to configure multiple hosts, while adding the metadaa works only for the `host` directive? Or is there something wrong with my config I'm not seeing? The documentation is very sparse in how this metadat is added.. Grtz Willem ---
**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\ **Post date:** [May 8, 2019, 1:26pm UTC](https://discuss.elastic.co/t/kubernetes-pods-found-but-all-metrics-0/180061/5 "2019-05-08T13:26:16Z")
Hi @willemdh, What's happening is that metadata enrichment is focused on a single node, as our deployment mode recommends un metricbeat instance per node in the cluster. I'm wondering, why did you go with a single instance? Best regards ---
**Author:** ![willemdh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/willemdh/32/16922_2.png) [@willemdh](https://discuss.elastic.co/u/willemdh)\ **Post date:** [May 8, 2019, 1:59pm UTC](https://discuss.elastic.co/t/kubernetes-pods-found-but-all-metrics-0/180061/6 "2019-05-08T13:59:29Z")
@exekias Short story =\> We started with running it as a daemonset in our Openshift cluster, but got too many issues. Even had to open a ticket with Red Hat, where Red Hat actually said running Beats as a daemonset is 'unsupported'. So we tried running it on 1 node where we ran into several authorization issues. the kube config sv account had to be adjusted, and also our master nodes are the only nodes which have the correct certificates (which are in /etc/origin/master/).. ssl.certificate: "crt" ssl.key: "key" Our Openshift admin didn't want to copy those certificates to the other (non-master) Openshift nodes. As it was allowed to configure multiple hosts, see the `hosts` directive, this seemed like a good idea. All metrics ware indexed correctly and we never really had the need for the extra metadata (for Metricbeat data, we did need it for Filebeat data) The metadata was added correctly but only on the node running the Metricbeat with the kubernetes module it seems now. ---
**Author:** ![exekias](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/exekias/32/28718_2.png) [@exekias](https://discuss.elastic.co/u/exekias)\ **Post date:** [May 8, 2019, 4:03pm UTC](https://discuss.elastic.co/t/kubernetes-pods-found-but-all-metrics-0/180061/7 "2019-05-08T16:03:04Z")
Thanks for the explanation, your current approach seems correct. We thought about adding a global scope to metadata processing & autodiscover, that feature may come in the future, that would simplify your settings. Br ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 5, 2019, 4:03pm UTC](https://discuss.elastic.co/t/kubernetes-pods-found-but-all-metrics-0/180061/8 "2019-06-05T16:03:06Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.