[ERROR] - Kubernetes Module - reading bearer token file

Elastic Stack Beats
1

Hi everyone.

I have Kubernetes 1.14.1, tried monitoring my cluster with Metricbeat 6.6.1 and Elasticsearch/Kibana. I used the follow configuration in kubernetes.yml:

- module: kubernetes
  metricsets:
    - node
    - system
    - pod
    - container
    - volume
  period: 10s
  hosts: ["https://kubernetes-master:10250"]
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  ssl.certificate_authorities:
    - /var/run/secrets/kubernetes.io/serviceaccount/service-ca.crt

but, i received the error below:

    INFO    instance/beat.go:281    Setup Beat: metricbeat; Version: 6.6.1
    INFO    add_cloud_metadata/add_cloud_metadata.go:319    add_cloud_metadata: hosting provider type not detected.
    INFO    elasticsearch/client.go:165     Elasticsearch url: http://elasticsearch:9200
    INFO    [publisher]     pipeline/module.go:110  Beat name: kubernetes-master
    INFO    [monitoring]    log/log.go:117  Starting metrics logging every 30s
    INFO    instance/beat.go:403    metricbeat start running.
    INFO    [monitoring]    log/log.go:152  Total non-zero metrics  {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":90,"time":{"ms":93}},"total":{"ticks":170,"time":{"ms":178},"value":170},"user":{"ticks":80,"time":{"ms":85}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":6},"info":{"ephemeral_id":"b947d375-e201-4b03-b628-2aa87de42297","uptime":{"ms":3097}},"memstats":{"gc_next":5542016,"memory_alloc":2976672,"memory_total":9877728,"rss":25583616}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"elasticsearch"},"pipeline":{"clients":0,"events":{"active":0}}},"system":{"cpu":{"cores":4},"load":{"1":0.19,"15":0.26,"5":0.2,"norm":{"1":0.0475,"15":0.065,"5":0.05}}}}}}
    INFO    [monitoring]    log/log.go:153  Uptime: 3.10495974s
    INFO    [monitoring]    log/log.go:130  Stopping metrics logging.
    INFO    instance/beat.go:413    metricbeat stopped.
    ERROR   instance/beat.go:911    Exiting: 1 error: 5 errors: reading bearer token file: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory; reading bearer token file: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory; reading bearer token file: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory; reading bearer token file: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory; reading bearer token file: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory

i saw many topics about this, already add the property ssl.verification_mode: none but without success. continue the same error, how can i solution it?

2

@grfneto Sorry I don't know kubernetes module besides using it with

- module: kubernetes
      metricsets:
        - node
        - system
        - pod
        - container
        - volume
      period: 10s
      host: ${NODE_NAME}
      hosts: ["localhost:10255"]

@exekias we need your help here :grimacing:

3

Hi @grfneto,

How are you running Metricbeat? We use this service account token to access the API, have a look to: https://kubernetes.io/docs/tasks/access-application-cluster/access-cluster/#accessing-the-api-from-a-pod.

If Metricbeat is correctly configured, I would expect that file to exist inside the pod, and contain the token to do authentication.

Best regards

4

hi @Kaiyan_Sheng I used the getting started with metricbeat option download rpm and i enabled the module kubernetes in the settings /etc/metricbeat/modules.d/kubernetes.yml file.

5

Hi @exekias the status of service is running but the log file show message of error and nothing arrived in the elasticsearch. :slightly_frowning_face:

6

Hi,

You can probably benefit from this guide: https://www.elastic.co/guide/en/beats/metricbeat/current/running-on-kubernetes.html

It shows how to run Metricbeat as a daemonset, so it gets run inside Kubernetes and gets access to the required APIs

7

Thank you @exekias

I did it this way.

best regards :wink:

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.