# Metricbeat kubernetes fails to collect

**URL:** <https://discuss.elastic.co/t/metricbeat-kubernetes-fails-to-collect/217414>\
**Category:** Beats\
**Tags:** metricbeat\
**Created:** [January 31, 2020, 3:40pm UTC](https://discuss.elastic.co/t/metricbeat-kubernetes-fails-to-collect/217414 "2020-01-31T15:40:16Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Babadofar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/babadofar/32/44803_2.png) [@Babadofar](https://discuss.elastic.co/u/Babadofar)\
**Post date:** [January 31, 2020, 3:40pm UTC](https://discuss.elastic.co/t/metricbeat-kubernetes-fails-to-collect/217414/1 "2020-01-31T15:40:16Z")

</div>

Using the latest elastic helm chart 7.5.2 on EKS kubernetes, I get these errors from metricbeat trying to fetch from kubernetes.volume

> 2020-01-31T15:32:52.119Z INFO module/wrapper.go:252 Error fetching data for metricset kubernetes.volume: error doing HTTP request to fetch 'volume' Metricset data: error making http request: Get [http://ip-192-168-80-40.eu-west-1.compute.internal:10255/stats/summary:](http://ip-192-168-80-40.eu-west-1.compute.internal:10255/stats/summary:) dial tcp 192.168.80.40:10255: connect: connection refused  
> 2020-01-31T15:32:53.595Z INFO module/wrapper.go:252 Error fetching data for metricset kubernetes.pod: error doing HTTP request to fetch 'pod' Metricset data: error making http request: Get [http://ip-192-168-80-40.eu-west-1.compute.internal:10255/stats/summary:](http://ip-192-168-80-40.eu-west-1.compute.internal:10255/stats/summary:) dial tcp 192.168.80.40:10255: connect: connection refused  
> 2020-01-31T15:32:57.509Z INFO module/wrapper.go:252 Error fetching data for metricset kubernetes.container: error doing HTTP request to fetch 'container' Metricset data: error making http request: Get [http://ip-192-168-80-40.eu-west-1.compute.internal:10255/stats/summary:](http://ip-192-168-80-40.eu-west-1.compute.internal:10255/stats/summary:) dial tcp 192.168.80.40:10255: connect: connection refused  
> 2020-01-31T15:32:57.805Z INFO module/wrapper.go:252 Error fetching data for metricset kubernetes.system: error doing HTTP request to fetch 'system' Metricset data: error making http request: Get [http://ip-192-168-80-40.eu-west-1.compute.internal:10255/stats/summary:](http://ip-192-168-80-40.eu-west-1.compute.internal:10255/stats/summary:) dial tcp 192.168.80.40:10255: connect: connection refused

---

<div class="post-metadata">

**Author:** ![pmercado](https://avatars.discourse-cdn.com/v4/letter/p/59ef9b/32.png) [@pmercado](https://discuss.elastic.co/u/pmercado)\
**Post date:** [February 1, 2020, 9:50pm UTC](https://discuss.elastic.co/t/metricbeat-kubernetes-fails-to-collect/217414/2 "2020-02-01T21:50:48Z")

</div>

You are trying to use the kubelet non authenticated port, which most kubernetes installations disable.

Can you try secure port 10250, adding certificates as needed?  
You can also log into the node and curl manually to make sure the kubelet endpoints are reachable.

---

<div class="post-metadata">

**Author:** ![Babadofar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/babadofar/32/44803_2.png) [@Babadofar](https://discuss.elastic.co/u/Babadofar)\
**Post date:** [February 8, 2020, 9:31pm UTC](https://discuss.elastic.co/t/metricbeat-kubernetes-fails-to-collect/217414/3 "2020-02-08T21:31:44Z")

</div>

Yes, you're right, that helped somewhat.

---

<div class="post-metadata">

**Author:** ![Babadofar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/babadofar/32/44803_2.png) [@Babadofar](https://discuss.elastic.co/u/Babadofar)\
**Post date:** [February 17, 2020, 8:31pm UTC](https://discuss.elastic.co/t/metricbeat-kubernetes-fails-to-collect/217414/4 "2020-02-17T20:31:51Z")

</div>

I get a 403 when requesting with the certificates. And I can't find these endpoints on the node. Kubernetes version 1.14

---

<div class="post-metadata">

**Author:** ![Babadofar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/babadofar/32/44803_2.png) [@Babadofar](https://discuss.elastic.co/u/Babadofar)\
**Post date:** [February 19, 2020, 2:39pm UTC](https://discuss.elastic.co/t/metricbeat-kubernetes-fails-to-collect/217414/5 "2020-02-19T14:39:50Z")

</div>

editing the clusterrole and adding `- nodes/stats` to the list of resources fixed this issue

---

<div class="post-metadata">

**Author:** ![Babadofar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/babadofar/32/44803_2.png) [@Babadofar](https://discuss.elastic.co/u/Babadofar)\
**Post date:** [February 19, 2020, 2:41pm UTC](https://discuss.elastic.co/t/metricbeat-kubernetes-fails-to-collect/217414/6 "2020-02-19T14:41:48Z")

</div>

In addition - settting the kubernetes config like this:

```
- module: kubernetes
  metricsets:
    - node
    - system
    - pod
    - volume
    - event
    - container
  host: "${NODE_NAME}"
  hosts: ["https://${NODE_NAME}:10250"]
  ssl.verification_mode: none
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
```

---

<div class="post-metadata">

**Author:** ![kasim123](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kasim123/32/58684_2.png) [@kasim123](https://discuss.elastic.co/u/kasim123)\
**Post date:** [March 10, 2020, 10:31am UTC](https://discuss.elastic.co/t/metricbeat-kubernetes-fails-to-collect/217414/7 "2020-03-10T10:31:42Z")

</div>

Hi @Babadofar,

Could you clearly explain what changes you had done clusterrole for metricbeat to work. could you share snippet of clusterrole.

Thanks,  
Kasim Shaik

---

<div class="post-metadata">

**Author:** ![Babadofar](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/babadofar/32/44803_2.png) [@Babadofar](https://discuss.elastic.co/u/Babadofar)\
**Post date:** [March 10, 2020, 12:05pm UTC](https://discuss.elastic.co/t/metricbeat-kubernetes-fails-to-collect/217414/8 "2020-03-10T12:05:56Z")

</div>

This is the entire clusterrole definition for metricbeat

```
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    jenkins.io/chart: env
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"rbac.authorization.k8s.io/v1beta1","kind":"ClusterRole","metadata":{"annotations":{"jenkins.io/chart":"env"},"labels":{"app":"jenkins-x-metricbeat","chart":"metricbeat-7.6.0","heritage":"Tiller","jenkins.io/chart-release":"jenkins-x","jenkins.io/version":"1","release":"jenkins-x"},"name":"jenkins-x-metricbeat-cluster-role"},"rules":[{"apiGroups":["extensions","apps",""],"resources":["namespaces","pods","events","deployments","nodes","replicasets"],"verbs":["get","list","watch"]}]}
  creationTimestamp: "2020-02-19T10:04:08Z"
  labels:
    app: jenkins-x-metricbeat
    chart: metricbeat-7.6.0
    heritage: Tiller
    jenkins.io/chart-release: jenkins-x
    jenkins.io/version: "1"
    release: jenkins-x
  name: jenkins-x-metricbeat-cluster-role
  resourceVersion: "23166250"
  selfLink: /apis/rbac.authorization.k8s.io/v1/clusterroles/jenkins-x-metricbeat-cluster-role
  uid: 2b52286d-52ff-11ea-860b-0293606def8a
rules:
- apiGroups:
  - extensions
  - apps
  - ""
  resources:
  - namespaces
  - pods
  - events
  - deployments
  - nodes
  - nodes/stats
  - replicasets
  verbs:
  - get
  - list
  - watch
```

---

<div class="post-metadata">

**Author:** ![kasim123](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kasim123/32/58684_2.png) [@kasim123](https://discuss.elastic.co/u/kasim123)\
**Post date:** [March 10, 2020, 12:08pm UTC](https://discuss.elastic.co/t/metricbeat-kubernetes-fails-to-collect/217414/9 "2020-03-10T12:08:55Z")

</div>

Thanks @Babadofar.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 7, 2020, 12:08pm UTC](https://discuss.elastic.co/t/metricbeat-kubernetes-fails-to-collect/217414/10 "2020-04-07T12:08:59Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.