Metricbeat kubernetes fails to collect

Elastic Stack Beats
1

Using the latest elastic helm chart 7.5.2 on EKS kubernetes, I get these errors from metricbeat trying to fetch from kubernetes.volume

2020-01-31T15:32:52.119Z INFO module/wrapper.go:252 Error fetching data for metricset kubernetes.volume: error doing HTTP request to fetch 'volume' Metricset data: error making http request: Get http://ip-192-168-80-40.eu-west-1.compute.internal:10255/stats/summary: dial tcp 192.168.80.40:10255: connect: connection refused
2020-01-31T15:32:53.595Z INFO module/wrapper.go:252 Error fetching data for metricset kubernetes.pod: error doing HTTP request to fetch 'pod' Metricset data: error making http request: Get http://ip-192-168-80-40.eu-west-1.compute.internal:10255/stats/summary: dial tcp 192.168.80.40:10255: connect: connection refused
2020-01-31T15:32:57.509Z INFO module/wrapper.go:252 Error fetching data for metricset kubernetes.container: error doing HTTP request to fetch 'container' Metricset data: error making http request: Get http://ip-192-168-80-40.eu-west-1.compute.internal:10255/stats/summary: dial tcp 192.168.80.40:10255: connect: connection refused
2020-01-31T15:32:57.805Z INFO module/wrapper.go:252 Error fetching data for metricset kubernetes.system: error doing HTTP request to fetch 'system' Metricset data: error making http request: Get http://ip-192-168-80-40.eu-west-1.compute.internal:10255/stats/summary: dial tcp 192.168.80.40:10255: connect: connection refused

2

You are trying to use the kubelet non authenticated port, which most kubernetes installations disable.

Can you try secure port 10250, adding certificates as needed?
You can also log into the node and curl manually to make sure the kubelet endpoints are reachable.

3

Yes, you're right, that helped somewhat.

4

I get a 403 when requesting with the certificates. And I can't find these endpoints on the node. Kubernetes version 1.14

5

editing the clusterrole and adding - nodes/stats to the list of resources fixed this issue

6

In addition - settting the kubernetes config like this:

- module: kubernetes
  metricsets:
    - node
    - system
    - pod
    - volume
    - event
    - container
  host: "${NODE_NAME}"
  hosts: ["https://${NODE_NAME}:10250"]
  ssl.verification_mode: none
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
1 Like
7

Hi @Babadofar,

Could you clearly explain what changes you had done clusterrole for metricbeat to work. could you share snippet of clusterrole.

Thanks,
Kasim Shaik

8

This is the entire clusterrole definition for metricbeat

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    jenkins.io/chart: env
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"rbac.authorization.k8s.io/v1beta1","kind":"ClusterRole","metadata":{"annotations":{"jenkins.io/chart":"env"},"labels":{"app":"jenkins-x-metricbeat","chart":"metricbeat-7.6.0","heritage":"Tiller","jenkins.io/chart-release":"jenkins-x","jenkins.io/version":"1","release":"jenkins-x"},"name":"jenkins-x-metricbeat-cluster-role"},"rules":[{"apiGroups":["extensions","apps",""],"resources":["namespaces","pods","events","deployments","nodes","replicasets"],"verbs":["get","list","watch"]}]}
  creationTimestamp: "2020-02-19T10:04:08Z"
  labels:
    app: jenkins-x-metricbeat
    chart: metricbeat-7.6.0
    heritage: Tiller
    jenkins.io/chart-release: jenkins-x
    jenkins.io/version: "1"
    release: jenkins-x
  name: jenkins-x-metricbeat-cluster-role
  resourceVersion: "23166250"
  selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/jenkins-x-metricbeat-cluster-role
  uid: 2b52286d-52ff-11ea-860b-0293606def8a
rules:
- apiGroups:
  - extensions
  - apps
  - ""
  resources:
  - namespaces
  - pods
  - events
  - deployments
  - nodes
  - nodes/stats
  - replicasets
  verbs:
  - get
  - list
  - watch
2 Likes
9

Thanks @Babadofar.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.