Problem in Metricbeat for kubernetes monitoring deployed by ECK

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

Hey,
In order to monitor my k8s cluster, I deployed Elastic, Kibana and Metricbeat by this yaml

In kibana dashboard I can see only metrics of the hosts (cluster nodes) but not metrics of kubernetes resources (Nodes, Pods, Containers, Volumes).

Metrics Hosts:

image
image1468×851 45.5 KB

Metrics Kubernetes:

image
image1732×810 35.9 KB

The metricbeat pod deployed on each node try fetch the metrics of pod, volumns, container etc.. but please see the pod log below:

2020-09-13T11:41:38.543Z        WARN    transport/tcp.go:53     DNS lookup failure "idit-kube-main": lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:41:38.543Z        INFO    module/wrapper.go:252   Error fetching data for metricset kubernetes.volume: error doing HTTP request to fetch 'volume' Metricset data: error making http request: Get https://idit-kube-main:10250/stats/summary: lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:41:39.482Z        WARN    transport/tcp.go:53     DNS lookup failure "idit-kube-main": lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:41:39.482Z        INFO    module/wrapper.go:252   Error fetching data for metricset kubernetes.pod: error doing HTTP request to fetch 'pod' Metricset data: error making http request: Get https://idit-kube-main:10250/stats/summary: lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:41:40.502Z        WARN    transport/tcp.go:53     DNS lookup failure "idit-kube-main": lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:41:40.503Z        INFO    module/wrapper.go:252   Error fetching data for metricset kubernetes.container: error doing HTTP request to fetch 'container' Metricset data: error making http request: Get https://idit-kube-main:10250/stats/summary: lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:41:41.707Z        WARN    transport/tcp.go:53     DNS lookup failure "idit-kube-main": lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:41:41.707Z        INFO    module/wrapper.go:252   Error fetching data for metricset kubernetes.node: error doing HTTP request to fetch 'node' Metricset data: error making http request: Get https://idit-kube-main:10250/stats/summary: lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:41:46.004Z        WARN    transport/tcp.go:53     DNS lookup failure "idit-kube-main": lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:41:46.004Z        INFO    module/wrapper.go:252   Error fetching data for metricset kubernetes.system: error doing HTTP request to fetch 'system' Metricset data: error making http request: Get https://idit-kube-main:10250/stats/summary: lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:41:48.542Z        WARN    transport/tcp.go:53     DNS lookup failure "idit-kube-main": lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:41:48.542Z        INFO    module/wrapper.go:252   Error fetching data for metricset kubernetes.volume: error doing HTTP request to fetch 'volume' Metricset data: error making http request: Get https://idit-kube-main:10250/stats/summary: lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:41:49.481Z        WARN    transport/tcp.go:53     DNS lookup failure "idit-kube-main": lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:41:49.481Z        INFO    module/wrapper.go:252   Error fetching data for metricset kubernetes.pod: error doing HTTP request to fetch 'pod' Metricset data: error making http request: Get https://idit-kube-main:10250/stats/summary: lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:41:50.502Z        WARN    transport/tcp.go:53     DNS lookup failure "idit-kube-main": lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:41:50.503Z        INFO    module/wrapper.go:252   Error fetching data for metricset kubernetes.container: error doing HTTP request to fetch 'container' Metricset data: error making http request: Get https://idit-kube-main:10250/stats/summary: lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:41:51.704Z        WARN    transport/tcp.go:53     DNS lookup failure "idit-kube-main": lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:41:51.704Z        INFO    module/wrapper.go:252   Error fetching data for metricset kubernetes.node: error doing HTTP request to fetch 'node' Metricset data: error making http request: Get https://idit-kube-main:10250/stats/summary: lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:41:56.003Z        WARN    transport/tcp.go:53     DNS lookup failure "idit-kube-main": lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:41:56.004Z        INFO    module/wrapper.go:252   Error fetching data for metricset kubernetes.system: error doing HTTP request to fetch 'system' Metricset data: error making http request: Get https://idit-kube-main:10250/stats/summary: lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:41:58.542Z        WARN    transport/tcp.go:53     DNS lookup failure "idit-kube-main": lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:41:58.542Z        INFO    module/wrapper.go:252   Error fetching data for metricset kubernetes.volume: error doing HTTP request to fetch 'volume' Metricset data: error making http request: Get https://idit-kube-main:10250/stats/summary: lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:41:59.481Z        WARN    transport/tcp.go:53     DNS lookup failure "idit-kube-main": lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:41:59.482Z        INFO    module/wrapper.go:252   Error fetching data for metricset kubernetes.pod: error doing HTTP request to fetch 'pod' Metricset data: error making http request: Get https://idit-kube-main:10250/stats/summary: lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:42:00.502Z        WARN    transport/tcp.go:53     DNS lookup failure "idit-kube-main": lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:42:00.503Z        INFO    module/wrapper.go:252   Error fetching data for metricset kubernetes.container: error doing HTTP request to fetch 'container' Metricset data: error making http request: Get https://idit-kube-main:10250/stats/summary: lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:42:01.704Z        WARN    transport/tcp.go:53     DNS lookup failure "idit-kube-main": lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:42:01.705Z        INFO    module/wrapper.go:252   Error fetching data for metricset kubernetes.node: error doing HTTP request to fetch 'node' Metricset data: error making http request: Get https://idit-kube-main:10250/stats/summary: lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:42:06.004Z        WARN    transport/tcp.go:53     DNS lookup failure "idit-kube-main": lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:42:06.004Z        INFO    module/wrapper.go:252   Error fetching data for metricset kubernetes.system: error doing HTTP request to fetch 'system' Metricset data: error making http request: Get https://idit-kube-main:10250/stats/summary: lookup idit-kube-main on 10.100.200.2:53: no such host
2020-09-13T11:42:07.193Z        INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":11330,"time":{"ms":318}},"total":{"ticks":28180,"time":{"ms":799},"value":28180},"user":{"ticks":1685

The idit-kube-main is name of the machine with installed kubectl that connect to remote cluster and i deployed the ELK by this machine to the remote cluster, but this machine isn't part of the cluster!
So why metricbeat consider this server as host in order to fetch the cluster metrics?

I use ELK version 7.5.2

Please Help

2

Hi @HadarPeeran,

Thanks for your question.

I'm not sure I understand the setup correctly - the logs you've pasted (with failing calls to https://idit-kube-main:10250/stats/summary) are coming from idit-kube-main host. Are you saying that it doesn't belong to your ES cluster? This is fine, as the yaml you've used deploys Metricbeat on all of the Kubernetes cluster nodes. Nevertheless, kubelet should be running there and Metricbeat should be able to access its endpoint.

If you are saying that this node is not part of you Kubernetes cluster then I'm not sure how Metricbeat could possibly be deployed there.

Did you make any changes to the yaml file you deployed? If there is an issue with choosing the right host name, can you try this file - https://raw.githubusercontent.com/elastic/cloud-on-k8s/master/config/recipes/beats/metricbeat_hosts.yaml? It's an updated version where node names are coming from downward API (see PR).

If that doesn't help, can you paste Metricbeat logs from other nodes and output of kubectl get -o yaml beat metricbeat?

3

thanks @dkow.

The problem was in below part of yaml:

image
image712×120 3.87 KB

I deployed the yaml by command: cat <<EOD | kubectl -f -
, it caused the $HOSTNAME to convert to local host name and deploy to all nodes with this static host name.
I tried deploy the yaml by file and not by above command and it worked:

image
image1506×898 55.8 KB

I have additional questions, Can you pleas help me:

  1. Does Metricbeat monitor the PV? Used/Free storage?
  2. Can you please explain why need deploy kube-state-metrics? Which metrics this pod exports?
  3. What is the different between use kubernetes module (as deployed by ECK) and Prometheus module?
4

Ah, nice catch. It's great to hear it works.

Regarding your questions:

  1. Does Metricbeat monitor the PV? Used/Free storage?

Yes. Look at kubernetes.volume.name, kubernetes.volume.fs.available.bytes and kubernetes.volume.fs.used.bytes in Discover with metricbeat-* template, like below:

Screenshot 2020-09-14 at 11.53.29
Screenshot 2020-09-14 at 11.53.292750×722 45 KB

  1. Can you please explain why need deploy kube-state-metrics? Which metrics this pod exports?

kube-state-metrics does have some overlap with other Kubernetes module metricssets. While I don't have an exhaustive diff on my hands you can take a look at the Kubernetes module docs. You can also compare data coming from built-in and kube-state-metrics for each of the types (pod in this case).

  1. What is the different between use kubernetes module (as deployed by ECK) and Prometheus module?

Prometheus module can be used to scrape any endpoint that exposes Prometheus metrics. Kubernetes module is geared towards Kubernetes metrics specifically and it knows which URL paths to hit for each resource. It does some transformations (renames) as well.

Btw, the Beats discuss forum might be better suited for questions like above as we tend to focus on the ECK itself.

1 Like
5

@dkow thank you so much.