Metricbeat vs k8s Clusterwide Stats Collection by a ServiceAccount

Attempting to grasp how to collect various stats from a k8s cluster as a newbie k8s admin...
Got a Linux host running a k8s cluster with a metricbeat running collecting various modules.d/system.yml datasets, would like this to also if possible to collect various cluster wide stats from k8s through modules.d/kubernetes.yml.

So I've created a new serviceaccount, fetch it's token: metricbeat like this:

kubectl create serviceaccount metricbeat
kubectl get secret `kubectl get serviceaccounts metricbeat -o yaml | awk '/^- name: /{print $3}'` -o yaml  | awk '/ token: /{print $2}' > /etc/metricbeat/sa.token

Set these into my kubernetes.yml:

- module: kubernetes
    - node
    - system
    - pod
    - container
  period: 1m
  hosts: ["https://localhost:6443"]
  bearer_token_file: /etc/metricbeat//sa.token

Only to get 401:

"error":{"message":"error doing HTTP request to fetch 'node' Metricset data: HTTP error 401 in : 401 Unauthorized"}

So I'm wondering what ClusterRole + Binding my ServiceAccount would need to have permission to collect the various cluster wide stats about pods,nodes,namespaces,events.

Found these samples here if running a metricbeat as pod on every cluster node, but not sure they'll cut it for me:

kind: ClusterRoleBinding
  name: metricbeat
- kind: ServiceAccount
  name: metricbeat
  namespace: kube-system
  kind: ClusterRole
  name: metricbeat

kind: ClusterRole
  name: metricbeat
    k8s-app: metricbeat
- apiGroups: [""]
  - nodes
  - namespaces
  - events
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups: ["extensions"]
  - replicasets
  verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
  - statefulsets
  - deployments
  verbs: ["get", "list", "watch"]
- apiGroups:
  - ""
  - nodes/stats
  - get

Any hints appreciated, TIA!

Hi @stefws!

Those metricsets (node, pod etc) collecting metrics from Kubelet API. So this is why you cannot access the metrics clusterwide. See

There are other Metricsets that can collect cluster wide metrics (retrieved from kube-state-metrics). Please have a look at the Module's documentation here, and let me know if you have questions!

Thanks, will look into State Metrics, but assume my service account still needs some permission/role assignment for some apiGroups/namespace(s) to be allowed to get such data?

Well, kube-state-metrics is like a side-car project which runs on k8s, collects metrics and exposes them in Prometheus format. So you will be collecting metrics from this project and not k8s itself.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.