Trouble Collecting Kubernetes apiserver Metricset

[rudolphk (Kurt Rudolph)]

I've been trying to setup metricbeat to monitor my k8s cluster. I've got my daemonset and pods working, but I'm having difficulty getting apiserver metrics. Originally I was getting ssl cert errors from metricbeat until I set the "ssl.certificate_authorities" option in my kubernetes module. Unfortunately I am now receiving nothing from the apiserver and no errors are showing up in the log. I suspect this is an RBAC issue because if I attempt to curl the /metrics endpoint I get an error.

{
"kind": "Status",
"apiVersion": "v1",
"metadata": {

},
"status": "Failure",
"message": "forbidden: User "system:anonymous" cannot get path "/metrics"",
"reason": "Forbidden",
"details": {

},
"code": 403
}

I have configured my daemonset and pod to use the metricbeat service account and I think that I've granted the appropriate permissions, however I must be missing something.

Here is the role config I'm binding to the service account

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: metricbeat
labels:
k8s-app: metricbeat
rules:

• apiGroups: [""]
  resources:
  • nodes
  • namespaces
  • events
  • pods
    verbs: ["get", "list", "watch"]
• apiGroups: ["extensions"]
  resources:
  • replicasets
    verbs: ["get", "list", "watch"]
• apiGroups: ["apps"]
  resources:
  • statefulsets
  • deployments
    verbs: ["get", "list", "watch"]
• apiGroups:
  • ""
    resources:
  • nodes/stats
  • nodes/metrics
    verbs:
  • get
• nonResourceURLs:
  • "/metrics"
    verbs:
  • get

Here is my pod deployment

---

Deploy singleton instance in the whole cluster for some unique data sources, like kube-state-metrics

apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: metricbeat
namespace: kube-system
labels:
k8s-app: metricbeat
spec:
template:
metadata:
labels:
k8s-app: metricbeat
spec:
serviceAccountName: metricbeat
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
containers:
- name: metricbeat
image: {{ .Values.beatImage }}
args: [
"-c", "/etc/metricbeat.yml",
"-e",
]
resources:
limits:
cpu: 200m
memory: 100Mi
requests:
cpu: 50m
memory: 50Mi
env:
- name: KAFKA_BROKERS
value: {{ .Values.kafka.brokerList }}
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
securityContext:
runAsUser: 0
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
volumeMounts:
- name: config
mountPath: /etc/metricbeat.yml
readOnly: true
subPath: metricbeat.yml
- name: modules
mountPath: /usr/share/metricbeat/modules.d
readOnly: true
volumes:
- name: config
configMap:
defaultMode: 0600
name: metricbeat-deployment-config
- name: modules
configMap:
defaultMode: 0600
name: metricbeat-deployment-modules

and my configmap

apiVersion: v1
kind: ConfigMap
metadata:
name: metricbeat-deployment-config
namespace: kube-system
labels:
k8s-app: metricbeat
data:
metricbeat.yml: |-
metricbeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
output.kafka:
hosts: ['${KAFKA_BROKERS}']
topic: {{ .Values.kafka.topic }}
required_acks: 1
compression: gzip
max_message_bytes: 1000000

apiVersion: v1
kind: ConfigMap
metadata:
name: metricbeat-deployment-modules
namespace: kube-system
labels:
k8s-app: metricbeat
data:
kubernetes.yml: |-
- module: kubernetes
enabled: true
metricsets:
- state_node
- state_deployment
- state_replicaset
- state_statefulset
- state_pod
- state_container
- event
period: 10s
host: ${NODE_NAME}
hosts: ["kube-state-metrics:8080"]
- module: kubernetes
enabled: true
metricsets:
- apiserver
#KUBERNETES_SERVICE_* env vars inherited automagically from cluster
hosts: ["https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}"]
#ssl.verification_mode: "none"
ssl.certificate_authorities: ["/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"]

[rudolphk (Kurt Rudolph)]

I figured it out. I was missing the bearer_token for the apiserver

- module: kubernetes
  enabled: true
  metricsets:
    - apiserver
  #KUBERNETES_SERVICE_* env vars inherited automagically from cluster
  hosts: ["https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}"]
  #ssl.verification_mode: "none"
  ssl.certificate_authorities: ["/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"]
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token