Kubernetes Module - Kubelet -unauthorized error

Elastic Stack Beats
1

Hi
I have Kubernetes 1.13.1 and Metricbeat 6.6.1 with Elasticsearch and Kibana. Not getting metrics of Kubelet though Kube-state-metric is working as expected.

--Not working--
kubernetes.yml: |-
- module: kubernetes
enabled: true
metricsets:
- node
- system
- pod
- container
- volume
period: 10s
host: {NODE_NAME} #hosts: ["https://{HOSTNAME}:10250"]
hosts: ["https://localhost:10250"]
ssl.verification_mode: none

I see below error in kibana

error.message:HTTP error 401 in container: 401 Unauthorized @timestamp:February 28th 2019, 21:05:53.407 beat.version:6.6.1 beat.name: beat.hostname: host.name: event.dataset:kubernetes.container event.duration:8,696,784 metricset.rtt:8,696 metricset.name:container metricset.module:kubernetes metricset.host:localhost:10250 _id:vMelN2kB8UQ-YSZ0AFq- _type:doc _index:metricbeat-6.6.1-2019.03.01 _score:
--------removed hostname in the above message----

Any help is appreciated - Thanks in advance

2

Hi @pbedadham and welcome :slight_smile:

An unauthorized error can mean that actually metricbeat is not being able to authenticate with kubelet. The proper configuration will depend on the configuration of your cluster.

If your kubelet has the plain http port open (defaults to 10255), just use:

    hosts: ["http://localhost:10255"]

If you want to use the secured port, you probably also need to provide some credentials. If you are using service accounts something like this should work:

    hosts: ["https://localhost:10250"]
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    ssl.certificate_authorities:
      - /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt
3

Thanks @jsoriano for reply. Made some changes as per below and it is working.

  hosts: ["https://${HOSTNAME}:10250"]
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  ssl:
    # disable verification since kubelete uses self signed certificate
    verification_mode: none
    certificate_authorities:
      - /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.