Metricbeat vs k8s Clusterwide Stats Collection by a ServiceAccount

Attempting to grasp how to collect various stats from a k8s cluster as a newbie k8s admin...
Got a Linux host running a k8s cluster with a metricbeat running collecting various modules.d/system.yml datasets, would like this to also if possible to collect various cluster wide stats from k8s through modules.d/kubernetes.yml.

So I've created a new serviceaccount, fetch it's token: metricbeat like this:

kubectl create serviceaccount metricbeat
kubectl get secret `kubectl get serviceaccounts metricbeat -o yaml | awk '/^- name: /{print $3}'` -o yaml  | awk '/ token: /{print $2}' > /etc/metricbeat/sa.token

Set these into my kubernetes.yml:

- module: kubernetes
  metricsets:
    - node
    - system
    - pod
    - container
  period: 1m
  hosts: ["https://localhost:6443"]
  bearer_token_file: /etc/metricbeat//sa.token

Only to get 401:

"error":{"message":"error doing HTTP request to fetch 'node' Metricset data: HTTP error 401 in : 401 Unauthorized"}

So I'm wondering what ClusterRole + Binding my ServiceAccount would need to have permission to collect the various cluster wide stats about pods,nodes,namespaces,events.

Found these samples here if running a metricbeat as pod on every cluster node, but not sure they'll cut it for me:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: metricbeat
subjects:
- kind: ServiceAccount
  name: metricbeat
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: metricbeat
  apiGroup: rbac.authorization.k8s.io


apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: metricbeat
  labels:
    k8s-app: metricbeat
rules:
- apiGroups: [""]
  resources:
  - nodes
  - namespaces
  - events
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups: ["extensions"]
  resources:
  - replicasets
  verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
  resources:
  - statefulsets
  - deployments
  verbs: ["get", "list", "watch"]
- apiGroups:
  - ""
  resources:
  - nodes/stats
  verbs:
  - get

Any hints appreciated, TIA!

Hi @stefws!

Those metricsets (node, pod etc) collecting metrics from Kubelet API. So this is why you cannot access the metrics clusterwide. See https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-kubernetes.html#_example_configuration_27.

There are other Metricsets that can collect cluster wide metrics (retrieved from kube-state-metrics). Please have a look at the Module's documentation here, and let me know if you have questions!

Thanks, will look into State Metrics, but assume my service account still needs some permission/role assignment for some apiGroups/namespace(s) to be allowed to get such data?

Well, kube-state-metrics is like a side-car project which runs on k8s, collects metrics and exposes them in Prometheus format. So you will be collecting metrics from this project and not k8s itself.