# Problem in Metricbeat for kubernetes monitoring deployed by ECK
**URL:** https://discuss.elastic.co/t/problem-in-metricbeat-for-kubernetes-monitoring-deployed-by-eck/248440
**Category:** Elastic Cloud on Kubernetes (ECK)
**Created:** [September 13, 2020, 12:10pm UTC](https://discuss.elastic.co/t/problem-in-metricbeat-for-kubernetes-monitoring-deployed-by-eck/248440 "2020-09-13T12:10:28Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@HadarPeeran](https://discuss.elastic.co/u/HadarPeeran)
#### Post date: [September 13, 2020, 12:10pm UTC](https://discuss.elastic.co/t/problem-in-metricbeat-for-kubernetes-monitoring-deployed-by-eck/248440/1 "2020-09-13T12:10:28Z")
Hey,
In order to monitor my k8s cluster, I deployed Elastic, Kibana and Metricbeat by [this yaml](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-beat-configuration-examples.html#k8s_metricbeat_for_kubernetes_monitoring)
In kibana dashboard I can see only metrics of the hosts (cluster nodes) but not metrics of kubernetes resources (Nodes, Pods, Containers, Volumes).
Metrics Hosts:
Metrics Kubernetes:
The metricbeat pod deployed on each node try fetch the metrics of pod, volumns, container etc.. but please see the pod log below:
> ```
> 2020-09-13T11:41:38.543Z WARN transport/tcp.go:53 DNS lookup failure "idit-kube-main": lookup idit-kube-main on 10.100.200.2:53: no such host
> 2020-09-13T11:41:38.543Z INFO module/wrapper.go:252 Error fetching data for metricset kubernetes.volume: error doing HTTP request to fetch 'volume' Metricset data: error making http request: Get https://idit-kube-main:10250/stats/summary: lookup idit-kube-main on 10.100.200.2:53: no such host
> 2020-09-13T11:41:39.482Z WARN transport/tcp.go:53 DNS lookup failure "idit-kube-main": lookup idit-kube-main on 10.100.200.2:53: no such host
> 2020-09-13T11:41:39.482Z INFO module/wrapper.go:252 Error fetching data for metricset kubernetes.pod: error doing HTTP request to fetch 'pod' Metricset data: error making http request: Get https://idit-kube-main:10250/stats/summary: lookup idit-kube-main on 10.100.200.2:53: no such host
> 2020-09-13T11:41:40.502Z WARN transport/tcp.go:53 DNS lookup failure "idit-kube-main": lookup idit-kube-main on 10.100.200.2:53: no such host
> 2020-09-13T11:41:40.503Z INFO module/wrapper.go:252 Error fetching data for metricset kubernetes.container: error doing HTTP request to fetch 'container' Metricset data: error making http request: Get https://idit-kube-main:10250/stats/summary: lookup idit-kube-main on 10.100.200.2:53: no such host
> 2020-09-13T11:41:41.707Z WARN transport/tcp.go:53 DNS lookup failure "idit-kube-main": lookup idit-kube-main on 10.100.200.2:53: no such host
> 2020-09-13T11:41:41.707Z INFO module/wrapper.go:252 Error fetching data for metricset kubernetes.node: error doing HTTP request to fetch 'node' Metricset data: error making http request: Get https://idit-kube-main:10250/stats/summary: lookup idit-kube-main on 10.100.200.2:53: no such host
> 2020-09-13T11:41:46.004Z WARN transport/tcp.go:53 DNS lookup failure "idit-kube-main": lookup idit-kube-main on 10.100.200.2:53: no such host
> 2020-09-13T11:41:46.004Z INFO module/wrapper.go:252 Error fetching data for metricset kubernetes.system: error doing HTTP request to fetch 'system' Metricset data: error making http request: Get https://idit-kube-main:10250/stats/summary: lookup idit-kube-main on 10.100.200.2:53: no such host
> 2020-09-13T11:41:48.542Z WARN transport/tcp.go:53 DNS lookup failure "idit-kube-main": lookup idit-kube-main on 10.100.200.2:53: no such host
> 2020-09-13T11:41:48.542Z INFO module/wrapper.go:252 Error fetching data for metricset kubernetes.volume: error doing HTTP request to fetch 'volume' Metricset data: error making http request: Get https://idit-kube-main:10250/stats/summary: lookup idit-kube-main on 10.100.200.2:53: no such host
> 2020-09-13T11:41:49.481Z WARN transport/tcp.go:53 DNS lookup failure "idit-kube-main": lookup idit-kube-main on 10.100.200.2:53: no such host
> 2020-09-13T11:41:49.481Z INFO module/wrapper.go:252 Error fetching data for metricset kubernetes.pod: error doing HTTP request to fetch 'pod' Metricset data: error making http request: Get https://idit-kube-main:10250/stats/summary: lookup idit-kube-main on 10.100.200.2:53: no such host
> 2020-09-13T11:41:50.502Z WARN transport/tcp.go:53 DNS lookup failure "idit-kube-main": lookup idit-kube-main on 10.100.200.2:53: no such host
> 2020-09-13T11:41:50.503Z INFO module/wrapper.go:252 Error fetching data for metricset kubernetes.container: error doing HTTP request to fetch 'container' Metricset data: error making http request: Get https://idit-kube-main:10250/stats/summary: lookup idit-kube-main on 10.100.200.2:53: no such host
> 2020-09-13T11:41:51.704Z WARN transport/tcp.go:53 DNS lookup failure "idit-kube-main": lookup idit-kube-main on 10.100.200.2:53: no such host
> 2020-09-13T11:41:51.704Z INFO module/wrapper.go:252 Error fetching data for metricset kubernetes.node: error doing HTTP request to fetch 'node' Metricset data: error making http request: Get https://idit-kube-main:10250/stats/summary: lookup idit-kube-main on 10.100.200.2:53: no such host
> 2020-09-13T11:41:56.003Z WARN transport/tcp.go:53 DNS lookup failure "idit-kube-main": lookup idit-kube-main on 10.100.200.2:53: no such host
> 2020-09-13T11:41:56.004Z INFO module/wrapper.go:252 Error fetching data for metricset kubernetes.system: error doing HTTP request to fetch 'system' Metricset data: error making http request: Get https://idit-kube-main:10250/stats/summary: lookup idit-kube-main on 10.100.200.2:53: no such host
> 2020-09-13T11:41:58.542Z WARN transport/tcp.go:53 DNS lookup failure "idit-kube-main": lookup idit-kube-main on 10.100.200.2:53: no such host
> 2020-09-13T11:41:58.542Z INFO module/wrapper.go:252 Error fetching data for metricset kubernetes.volume: error doing HTTP request to fetch 'volume' Metricset data: error making http request: Get https://idit-kube-main:10250/stats/summary: lookup idit-kube-main on 10.100.200.2:53: no such host
> 2020-09-13T11:41:59.481Z WARN transport/tcp.go:53 DNS lookup failure "idit-kube-main": lookup idit-kube-main on 10.100.200.2:53: no such host
> 2020-09-13T11:41:59.482Z INFO module/wrapper.go:252 Error fetching data for metricset kubernetes.pod: error doing HTTP request to fetch 'pod' Metricset data: error making http request: Get https://idit-kube-main:10250/stats/summary: lookup idit-kube-main on 10.100.200.2:53: no such host
> 2020-09-13T11:42:00.502Z WARN transport/tcp.go:53 DNS lookup failure "idit-kube-main": lookup idit-kube-main on 10.100.200.2:53: no such host
> 2020-09-13T11:42:00.503Z INFO module/wrapper.go:252 Error fetching data for metricset kubernetes.container: error doing HTTP request to fetch 'container' Metricset data: error making http request: Get https://idit-kube-main:10250/stats/summary: lookup idit-kube-main on 10.100.200.2:53: no such host
> 2020-09-13T11:42:01.704Z WARN transport/tcp.go:53 DNS lookup failure "idit-kube-main": lookup idit-kube-main on 10.100.200.2:53: no such host
> 2020-09-13T11:42:01.705Z INFO module/wrapper.go:252 Error fetching data for metricset kubernetes.node: error doing HTTP request to fetch 'node' Metricset data: error making http request: Get https://idit-kube-main:10250/stats/summary: lookup idit-kube-main on 10.100.200.2:53: no such host
> 2020-09-13T11:42:06.004Z WARN transport/tcp.go:53 DNS lookup failure "idit-kube-main": lookup idit-kube-main on 10.100.200.2:53: no such host
> 2020-09-13T11:42:06.004Z INFO module/wrapper.go:252 Error fetching data for metricset kubernetes.system: error doing HTTP request to fetch 'system' Metricset data: error making http request: Get https://idit-kube-main:10250/stats/summary: lookup idit-kube-main on 10.100.200.2:53: no such host
> 2020-09-13T11:42:07.193Z INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":11330,"time":{"ms":318}},"total":{"ticks":28180,"time":{"ms":799},"value":28180},"user":{"ticks":1685
>
> ```
The **idit-kube-main** is name of the machine with installed kubectl that connect to remote cluster and i deployed the ELK by this machine to the remote cluster, but this machine **isn't part** of the cluster!
So why metricbeat consider this server as host in order to fetch the cluster metrics?
**I use ELK version 7.5.2**
Please Help
---
### Author:  [@dkow](https://discuss.elastic.co/u/dkow)
#### Post date: [September 14, 2020, 5:04am UTC](https://discuss.elastic.co/t/problem-in-metricbeat-for-kubernetes-monitoring-deployed-by-eck/248440/2 "2020-09-14T05:04:56Z")
Hi @HadarPeeran,
Thanks for your question.
I'm not sure I understand the setup correctly - the logs you've pasted (with failing calls to `https://idit-kube-main:10250/stats/summary`) are coming from `idit-kube-main` host. Are you saying that it doesn't belong to your ES cluster? This is fine, as the `yaml` you've used deploys Metricbeat on all of the Kubernetes cluster nodes. Nevertheless, kubelet should be running there and Metricbeat should be able to access its endpoint.
If you are saying that this node is not part of you Kubernetes cluster then I'm not sure how Metricbeat could possibly be deployed there.
Did you make any changes to the yaml file you deployed? If there is an issue with choosing the right host name, can you try this file - `https://raw.githubusercontent.com/elastic/cloud-on-k8s/master/config/recipes/beats/metricbeat_hosts.yaml`? It's an updated version where node names are coming from downward API (see [PR](https://github.com/elastic/cloud-on-k8s/pull/3724)).
If that doesn't help, can you paste Metricbeat logs from other nodes and output of `kubectl get -o yaml beat metricbeat`?
---
### Author:  [@HadarPeeran](https://discuss.elastic.co/u/HadarPeeran)
#### Post date: [September 14, 2020, 7:34am UTC](https://discuss.elastic.co/t/problem-in-metricbeat-for-kubernetes-monitoring-deployed-by-eck/248440/3 "2020-09-14T07:34:33Z")
> [@dkow](#):
>
> kubectl get -o yaml beat metricbeat
thanks @dkow.
The problem was in below part of yaml:
I deployed the yaml by command: `cat <
### Author:  [@dkow](https://discuss.elastic.co/u/dkow)
#### Post date: [September 14, 2020, 10:14am UTC](https://discuss.elastic.co/t/problem-in-metricbeat-for-kubernetes-monitoring-deployed-by-eck/248440/4 "2020-09-14T10:14:12Z")
Ah, nice catch. It's great to hear it works.
Regarding your questions:
> 1. Does Metricbeat monitor the PV? Used/Free storage?
Yes. Look at `kubernetes.volume.name`, `kubernetes.volume.fs.available.bytes` and `kubernetes.volume.fs.used.bytes` in Discover with `metricbeat-*` template, like below:
> 1. Can you please explain why need deploy kube-state-metrics? Which metrics this pod exports?
`kube-state-metrics` does have some overlap with other Kubernetes module metricssets. While I don't have an exhaustive diff on my hands you can take a look at the Kubernetes module [docs](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-kubernetes.html). You can also compare data coming from [built-in](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-metricset-kubernetes-pod.html) and [kube-state-metrics](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-metricset-kubernetes-state_pod.html) for each of the types (pod in this case).
> 1. What is the different between use kubernetes module (as deployed by ECK) and Prometheus module?
Prometheus [module](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-prometheus.html) can be used to scrape any endpoint that exposes Prometheus metrics. Kubernetes [module](https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-kubernetes.html) is geared towards Kubernetes metrics specifically and it knows which URL paths to hit for each resource. It does some transformations (renames) as well.
Btw, the [Beats discuss forum](https://discuss.elastic.co/c/elastic-stack/beats/28) might be better suited for questions like above as we tend to focus on the ECK itself.
---
### Author:  [@HadarPeeran](https://discuss.elastic.co/u/HadarPeeran)
#### Post date: [September 14, 2020, 7:46pm UTC](https://discuss.elastic.co/t/problem-in-metricbeat-for-kubernetes-monitoring-deployed-by-eck/248440/5 "2020-09-14T19:46:43Z")
@dkow thank you so much.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [November 4, 2022, 8:13am UTC](https://discuss.elastic.co/t/problem-in-metricbeat-for-kubernetes-monitoring-deployed-by-eck/248440/6 "2022-11-04T08:13:02Z")