Metricbeat vs k8s Clusterwide Stats Collection by a ServiceAccount

Attempting to grasp how to collect various stats from a k8s cluster as a newbie k8s admin...
Got a Linux host running a k8s cluster with a metricbeat running collecting various modules.d/system.yml datasets, would like this to also if possible to collect various cluster wide stats from k8s through modules.d/kubernetes.yml.

So I've created a new serviceaccount, fetch it's token: metricbeat like this:

kubectl create serviceaccount metricbeat
kubectl get secret `kubectl get serviceaccounts metricbeat -o yaml | awk '/^- name: /{print $3}'` -o yaml  | awk '/ token: /{print $2}' > /etc/metricbeat/sa.token

Set these into my kubernetes.yml:

- module: kubernetes
  metricsets:
    - node
    - system
    - pod
    - container
  period: 1m
  hosts: ["https://localhost:6443"]
  bearer_token_file: /etc/metricbeat//sa.token

Only to get 401:

"error":{"message":"error doing HTTP request to fetch 'node' Metricset data: HTTP error 401 in : 401 Unauthorized"}

So I'm wondering what ClusterRole + Binding my ServiceAccount would need to have permission to collect the various cluster wide stats about pods,nodes,namespaces,events.

Found these samples here if running a metricbeat as pod on every cluster node, but not sure they'll cut it for me:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: metricbeat
subjects:
- kind: ServiceAccount
  name: metricbeat
  namespace: kube-system
roleRef:
  kind: ClusterRole
  name: metricbeat
  apiGroup: rbac.authorization.k8s.io


apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: metricbeat
  labels:
    k8s-app: metricbeat
rules:
- apiGroups: [""]
  resources:
  - nodes
  - namespaces
  - events
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups: ["extensions"]
  resources:
  - replicasets
  verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
  resources:
  - statefulsets
  - deployments
  verbs: ["get", "list", "watch"]
- apiGroups:
  - ""
  resources:
  - nodes/stats
  verbs:
  - get

Any hints appreciated, TIA!

Hi @stefws!

Those metricsets (node, pod etc) collecting metrics from Kubelet API. So this is why you cannot access the metrics clusterwide. See https://www.elastic.co/guide/en/beats/metricbeat/current/metricbeat-module-kubernetes.html#_example_configuration_27.

There are other Metricsets that can collect cluster wide metrics (retrieved from kube-state-metrics). Please have a look at the Module's documentation here, and let me know if you have questions!

Thanks, will look into State Metrics, but assume my service account still needs some permission/role assignment for some apiGroups/namespace(s) to be allowed to get such data?

Well, kube-state-metrics is like a side-car project which runs on k8s, collects metrics and exposes them in Prometheus format. So you will be collecting metrics from this project and not k8s itself.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.