# Trouble Collecting Kubernetes apiserver Metricset
**URL:** https://discuss.elastic.co/t/trouble-collecting-kubernetes-apiserver-metricset/186529
**Category:** Beats
**Tags:** metricbeat
**Created:** [June 19, 2019, 6:23pm UTC](https://discuss.elastic.co/t/trouble-collecting-kubernetes-apiserver-metricset/186529 "2019-06-19T18:23:43Z")
**Posts on this page:** 3
**Page:** 1
### Author:  [@rudolphk](https://discuss.elastic.co/u/rudolphk)
#### Post date: [June 19, 2019, 6:23pm UTC](https://discuss.elastic.co/t/trouble-collecting-kubernetes-apiserver-metricset/186529/1 "2019-06-19T18:23:43Z")
I've been trying to setup metricbeat to monitor my k8s cluster. I've got my daemonset and pods working, but I'm having difficulty getting apiserver metrics. Originally I was getting ssl cert errors from metricbeat until I set the "ssl.certificate\_authorities" option in my kubernetes module. Unfortunately I am now receiving nothing from the apiserver and no errors are showing up in the log. I suspect this is an RBAC issue because if I attempt to curl the /metrics endpoint I get an error.
> {
> "kind": "Status",
> "apiVersion": "v1",
> "metadata": {
>
> },
> "status": "Failure",
> "message": "forbidden: User "system:anonymous" cannot get path "/metrics"",
> "reason": "Forbidden",
> "details": {
>
> },
> "code": 403
> }
I have configured my daemonset and pod to use the metricbeat service account and I _think_ that I've granted the appropriate permissions, however I must be missing something.
Here is the role config I'm binding to the service account
> apiVersion: [rbac.authorization.k8s.io/v1beta1](http://rbac.authorization.k8s.io/v1beta1)
> kind: ClusterRole
> metadata:
> name: metricbeat
> labels:
> k8s-app: metricbeat
> rules:
>
> - apiGroups: [""]
> resources:
> - nodes
> - namespaces
> - events
> - pods
> verbs: ["get", "list", "watch"]
>
> - apiGroups: ["extensions"]
> resources:
> - replicasets
> verbs: ["get", "list", "watch"]
>
> - apiGroups: ["apps"]
> resources:
> - statefulsets
> - deployments
> verbs: ["get", "list", "watch"]
>
> - apiGroups:
> - ""
> resources:
> - nodes/stats
> - nodes/metrics
> verbs:
> - get
>
> - nonResourceURLs:
> - "/metrics"
> verbs:
> - get
Here is my pod deployment
> * * *
>
> # Deploy singleton instance in the whole cluster for some unique data sources, like kube-state-metrics
>
> apiVersion: apps/v1beta1
> kind: Deployment
> metadata:
> name: metricbeat
> namespace: kube-system
> labels:
> k8s-app: metricbeat
> spec:
> template:
> metadata:
> labels:
> k8s-app: metricbeat
> spec:
> serviceAccountName: metricbeat
> hostNetwork: true
> dnsPolicy: ClusterFirstWithHostNet
> containers:
> - name: metricbeat
> image: {{ .Values.beatImage }}
> args: [
> "-c", "/etc/metricbeat.yml",
> "-e",
> ]
> resources:
> limits:
> cpu: 200m
> memory: 100Mi
> requests:
> cpu: 50m
> memory: 50Mi
> env:
> - name: KAFKA\_BROKERS
> value: {{ .Values.kafka.brokerList }}
> - name: NODE\_NAME
> valueFrom:
> fieldRef:
> fieldPath: spec.nodeName
> securityContext:
> runAsUser: 0
> resources:
> limits:
> memory: 200Mi
> requests:
> cpu: 100m
> memory: 100Mi
> volumeMounts:
> - name: config
> mountPath: /etc/metricbeat.yml
> readOnly: true
> subPath: metricbeat.yml
> - name: modules
> mountPath: /usr/share/metricbeat/modules.d
> readOnly: true
> volumes:
> - name: config
> configMap:
> defaultMode: 0600
> name: metricbeat-deployment-config
> - name: modules
> configMap:
> defaultMode: 0600
> name: metricbeat-deployment-modules
and my configmap
> ## apiVersion: v1 kind: ConfigMap metadata: name: metricbeat-deployment-config namespace: kube-system labels: k8s-app: metricbeat data: metricbeat.yml: |- metricbeat.config.modules: path: ${path.config}/modules.d/\*.yml reload.enabled: false output.kafka: hosts: ['${KAFKA\_BROKERS}'] topic: {{ .Values.kafka.topic }} required\_acks: 1 compression: gzip max\_message\_bytes: 1000000
>
> apiVersion: v1
> kind: ConfigMap
> metadata:
> name: metricbeat-deployment-modules
> namespace: kube-system
> labels:
> k8s-app: metricbeat
> data:
> kubernetes.yml: |-
> - module: kubernetes
> enabled: true
> metricsets:
> - state\_node
> - state\_deployment
> - state\_replicaset
> - state\_statefulset
> - state\_pod
> - state\_container
> - event
> period: 10s
> host: ${NODE\_NAME}
> hosts: ["kube-state-metrics:8080"]
> - module: kubernetes
> enabled: true
> metricsets:
> - apiserver
> #KUBERNETES\_SERVICE\_\* env vars inherited automagically from cluster
> hosts: ["https://${KUBERNETES\_SERVICE\_HOST}:${KUBERNETES\_SERVICE\_PORT}"]
> #ssl.verification\_mode: "none"
> ssl.certificate\_authorities: ["/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"]
---
### Author:  [@rudolphk](https://discuss.elastic.co/u/rudolphk)
#### Post date: [June 20, 2019, 1:23pm UTC](https://discuss.elastic.co/t/trouble-collecting-kubernetes-apiserver-metricset/186529/2 "2019-06-20T13:23:27Z")
I figured it out. I was missing the bearer\_token for the apiserver
> ```
> - module: kubernetes
> enabled: true
> metricsets:
> - apiserver
> #KUBERNETES_SERVICE_* env vars inherited automagically from cluster
> hosts: ["https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}"]
> #ssl.verification_mode: "none"
> ssl.certificate_authorities: ["/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"]
> bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
>
> ```
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 18, 2019, 1:23pm UTC](https://discuss.elastic.co/t/trouble-collecting-kubernetes-apiserver-metricset/186529/3 "2019-07-18T13:23:30Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.