Attempting to grasp how to collect various stats from a k8s cluster as a newbie k8s admin...
Got a Linux host running a k8s cluster with a metricbeat running collecting various modules.d/system.yml datasets, would like this to also if possible to collect various cluster wide stats from k8s through modules.d/kubernetes.yml.
So I've created a new serviceaccount, fetch it's token: metricbeat like this:
kubectl create serviceaccount metricbeat
kubectl get secret `kubectl get serviceaccounts metricbeat -o yaml | awk '/^- name: /{print $3}'` -o yaml | awk '/ token: /{print $2}' > /etc/metricbeat/sa.token
Set these into my kubernetes.yml:
- module: kubernetes
metricsets:
- node
- system
- pod
- container
period: 1m
hosts: ["https://localhost:6443"]
bearer_token_file: /etc/metricbeat//sa.token
Only to get 401:
"error":{"message":"error doing HTTP request to fetch 'node' Metricset data: HTTP error 401 in : 401 Unauthorized"}
So I'm wondering what ClusterRole + Binding my ServiceAccount would need to have permission to collect the various cluster wide stats about pods,nodes,namespaces,events.
Found these samples here if running a metricbeat as pod on every cluster node, but not sure they'll cut it for me:
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: metricbeat
subjects:
- kind: ServiceAccount
name: metricbeat
namespace: kube-system
roleRef:
kind: ClusterRole
name: metricbeat
apiGroup: rbac.authorization.k8s.io
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: metricbeat
labels:
k8s-app: metricbeat
rules:
- apiGroups: [""]
resources:
- nodes
- namespaces
- events
- pods
verbs: ["get", "list", "watch"]
- apiGroups: ["extensions"]
resources:
- replicasets
verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
resources:
- statefulsets
- deployments
verbs: ["get", "list", "watch"]
- apiGroups:
- ""
resources:
- nodes/stats
verbs:
- get
Any hints appreciated, TIA!