Problem in Metricbeat for kubernetes monitoring deployed by ECK

HadarPeeran · September 13, 2020, 12:10pm

Hey,
In order to monitor my k8s cluster, I deployed Elastic, Kibana and Metricbeat by this yaml

In kibana dashboard I can see only metrics of the hosts (cluster nodes) but not metrics of kubernetes resources (Nodes, Pods, Containers, Volumes).

Metrics Hosts:

Metrics Kubernetes:

The metricbeat pod deployed on each node try fetch the metrics of pod, volumns, container etc.. but please see the pod log below:

2020-09-13T11:41:38.543Z        WARN    transport/tcp.go:53     DNS lookup failure "idit-kube-main": lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:41:38.543Z        INFO    module/wrapper.go:252   Error fetching data for metricset kubernetes.volume: error doing HTTP request to fetch 'volume' Metricset data: error making http request: Get https://idit-kube-main:10250/stats/summary: lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:41:39.482Z        WARN    transport/tcp.go:53     DNS lookup failure "idit-kube-main": lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:41:39.482Z        INFO    module/wrapper.go:252   Error fetching data for metricset kubernetes.pod: error doing HTTP request to fetch 'pod' Metricset data: error making http request: Get https://idit-kube-main:10250/stats/summary: lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:41:40.502Z        WARN    transport/tcp.go:53     DNS lookup failure "idit-kube-main": lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:41:40.503Z        INFO    module/wrapper.go:252   Error fetching data for metricset kubernetes.container: error doing HTTP request to fetch 'container' Metricset data: error making http request: Get https://idit-kube-main:10250/stats/summary: lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:41:41.707Z        WARN    transport/tcp.go:53     DNS lookup failure "idit-kube-main": lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:41:41.707Z        INFO    module/wrapper.go:252   Error fetching data for metricset kubernetes.node: error doing HTTP request to fetch 'node' Metricset data: error making http request: Get https://idit-kube-main:10250/stats/summary: lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:41:46.004Z        WARN    transport/tcp.go:53     DNS lookup failure "idit-kube-main": lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:41:46.004Z        INFO    module/wrapper.go:252   Error fetching data for metricset kubernetes.system: error doing HTTP request to fetch 'system' Metricset data: error making http request: Get https://idit-kube-main:10250/stats/summary: lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:41:48.542Z        WARN    transport/tcp.go:53     DNS lookup failure "idit-kube-main": lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:41:48.542Z        INFO    module/wrapper.go:252   Error fetching data for metricset kubernetes.volume: error doing HTTP request to fetch 'volume' Metricset data: error making http request: Get https://idit-kube-main:10250/stats/summary: lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:41:49.481Z        WARN    transport/tcp.go:53     DNS lookup failure "idit-kube-main": lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:41:49.481Z        INFO    module/wrapper.go:252   Error fetching data for metricset kubernetes.pod: error doing HTTP request to fetch 'pod' Metricset data: error making http request: Get https://idit-kube-main:10250/stats/summary: lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:41:50.502Z        WARN    transport/tcp.go:53     DNS lookup failure "idit-kube-main": lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:41:50.503Z        INFO    module/wrapper.go:252   Error fetching data for metricset kubernetes.container: error doing HTTP request to fetch 'container' Metricset data: error making http request: Get https://idit-kube-main:10250/stats/summary: lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:41:51.704Z        WARN    transport/tcp.go:53     DNS lookup failure "idit-kube-main": lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:41:51.704Z        INFO    module/wrapper.go:252   Error fetching data for metricset kubernetes.node: error doing HTTP request to fetch 'node' Metricset data: error making http request: Get https://idit-kube-main:10250/stats/summary: lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:41:56.003Z        WARN    transport/tcp.go:53     DNS lookup failure "idit-kube-main": lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:41:56.004Z        INFO    module/wrapper.go:252   Error fetching data for metricset kubernetes.system: error doing HTTP request to fetch 'system' Metricset data: error making http request: Get https://idit-kube-main:10250/stats/summary: lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:41:58.542Z        WARN    transport/tcp.go:53     DNS lookup failure "idit-kube-main": lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:41:58.542Z        INFO    module/wrapper.go:252   Error fetching data for metricset kubernetes.volume: error doing HTTP request to fetch 'volume' Metricset data: error making http request: Get https://idit-kube-main:10250/stats/summary: lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:41:59.481Z        WARN    transport/tcp.go:53     DNS lookup failure "idit-kube-main": lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:41:59.482Z        INFO    module/wrapper.go:252   Error fetching data for metricset kubernetes.pod: error doing HTTP request to fetch 'pod' Metricset data: error making http request: Get https://idit-kube-main:10250/stats/summary: lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:42:00.502Z        WARN    transport/tcp.go:53     DNS lookup failure "idit-kube-main": lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:42:00.503Z        INFO    module/wrapper.go:252   Error fetching data for metricset kubernetes.container: error doing HTTP request to fetch 'container' Metricset data: error making http request: Get https://idit-kube-main:10250/stats/summary: lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:42:01.704Z        WARN    transport/tcp.go:53     DNS lookup failure "idit-kube-main": lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:42:01.705Z        INFO    module/wrapper.go:252   Error fetching data for metricset kubernetes.node: error doing HTTP request to fetch 'node' Metricset data: error making http request: Get https://idit-kube-main:10250/stats/summary: lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:42:06.004Z        WARN    transport/tcp.go:53     DNS lookup failure "idit-kube-main": lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:42:06.004Z        INFO    module/wrapper.go:252   Error fetching data for metricset kubernetes.system: error doing HTTP request to fetch 'system' Metricset data: error making http request: Get https://idit-kube-main:10250/stats/summary: lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:42:07.193Z        INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":11330,"time":{"ms":318}},"total":{"ticks":28180,"time":{"ms":799},"value":28180},"user":{"ticks":1685

The idit-kube-main is name of the machine with installed kubectl that connect to remote cluster and i deployed the ELK by this machine to the remote cluster, but this machine isn't part of the cluster!
So why metricbeat consider this server as host in order to fetch the cluster metrics?

I use ELK version 7.5.2

Please Help

dkow · September 14, 2020, 5:04am

Hi @HadarPeeran,

Thanks for your question.

I'm not sure I understand the setup correctly - the logs you've pasted (with failing calls to https://idit-kube-main:10250/stats/summary) are coming from idit-kube-main host. Are you saying that it doesn't belong to your ES cluster? This is fine, as the yaml you've used deploys Metricbeat on all of the Kubernetes cluster nodes. Nevertheless, kubelet should be running there and Metricbeat should be able to access its endpoint.

If you are saying that this node is not part of you Kubernetes cluster then I'm not sure how Metricbeat could possibly be deployed there.

Did you make any changes to the yaml file you deployed? If there is an issue with choosing the right host name, can you try this file - https://raw.githubusercontent.com/elastic/cloud-on-k8s/master/config/recipes/beats/metricbeat_hosts.yaml? It's an updated version where node names are coming from downward API (see PR).

If that doesn't help, can you paste Metricbeat logs from other nodes and output of kubectl get -o yaml beat metricbeat?

HadarPeeran · September 14, 2020, 7:34am

thanks @dkow.

The problem was in below part of yaml:

I deployed the yaml by command: cat <<EOD | kubectl -f -
, it caused the $HOSTNAME to convert to local host name and deploy to all nodes with this static host name.
I tried deploy the yaml by file and not by above command and it worked:

I have additional questions, Can you pleas help me:

Does Metricbeat monitor the PV? Used/Free storage?
Can you please explain why need deploy kube-state-metrics? Which metrics this pod exports?
What is the different between use kubernetes module (as deployed by ECK) and Prometheus module?

dkow · September 14, 2020, 10:14am

Ah, nice catch. It's great to hear it works.

Regarding your questions:

Does Metricbeat monitor the PV? Used/Free storage?

Yes. Look at kubernetes.volume.name, kubernetes.volume.fs.available.bytes and kubernetes.volume.fs.used.bytes in Discover with metricbeat-* template, like below:

Can you please explain why need deploy kube-state-metrics? Which metrics this pod exports?

kube-state-metrics does have some overlap with other Kubernetes module metricssets. While I don't have an exhaustive diff on my hands you can take a look at the Kubernetes module docs. You can also compare data coming from built-in and kube-state-metrics for each of the types (pod in this case).

What is the different between use kubernetes module (as deployed by ECK) and Prometheus module?

Prometheus module can be used to scrape any endpoint that exposes Prometheus metrics. Kubernetes module is geared towards Kubernetes metrics specifically and it knows which URL paths to hit for each resource. It does some transformations (renames) as well.

Btw, the Beats discuss forum might be better suited for questions like above as we tend to focus on the ECK itself.

HadarPeeran · September 14, 2020, 7:46pm

@dkow thank you so much.

Topic		Replies	Views
Problem in Metricbeat for kubernetes monitoring deployed by ECK Elastic Cloud on Kubernetes (ECK)	1	372	March 8, 2023
Elastic_agent.metricbeat error fetching data from metricsets.kubernetes.system Elastic Agent metricbeat	2	161	April 29, 2024
Install Metricbeat on Kubernetes Cluster ( Azure) Beats metricbeat	6	554	February 17, 2023
Error making http request Elastic Agent metricbeat	1	263	February 9, 2023
Metricbeat kubernetes fails to collect Beats metricbeat	9	5740	April 7, 2020

Problem in Metricbeat for kubernetes monitoring deployed by ECK

Related Topics