Trouble Collecting Kubernetes apiserver Metricset

I've been trying to setup metricbeat to monitor my k8s cluster. I've got my daemonset and pods working, but I'm having difficulty getting apiserver metrics. Originally I was getting ssl cert errors from metricbeat until I set the "ssl.certificate_authorities" option in my kubernetes module. Unfortunately I am now receiving nothing from the apiserver and no errors are showing up in the log. I suspect this is an RBAC issue because if I attempt to curl the /metrics endpoint I get an error.

{
"kind": "Status",
"apiVersion": "v1",
"metadata": {

},
"status": "Failure",
"message": "forbidden: User "system:anonymous" cannot get path "/metrics"",
"reason": "Forbidden",
"details": {

},
"code": 403
}

I have configured my daemonset and pod to use the metricbeat service account and I think that I've granted the appropriate permissions, however I must be missing something.

Here is the role config I'm binding to the service account

apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: metricbeat
labels:
k8s-app: metricbeat
rules:

  • apiGroups: [""]
    resources:
    • nodes
    • namespaces
    • events
    • pods
      verbs: ["get", "list", "watch"]
  • apiGroups: ["extensions"]
    resources:
    • replicasets
      verbs: ["get", "list", "watch"]
  • apiGroups: ["apps"]
    resources:
    • statefulsets
    • deployments
      verbs: ["get", "list", "watch"]
  • apiGroups:
    • ""
      resources:
    • nodes/stats
    • nodes/metrics
      verbs:
    • get
  • nonResourceURLs:
    • "/metrics"
      verbs:
    • get

Here is my pod deployment


Deploy singleton instance in the whole cluster for some unique data sources, like kube-state-metrics

apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: metricbeat
namespace: kube-system
labels:
k8s-app: metricbeat
spec:
template:
metadata:
labels:
k8s-app: metricbeat
spec:
serviceAccountName: metricbeat
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
containers:
- name: metricbeat
image: {{ .Values.beatImage }}
args: [
"-c", "/etc/metricbeat.yml",
"-e",
]
resources:
limits:
cpu: 200m
memory: 100Mi
requests:
cpu: 50m
memory: 50Mi
env:
- name: KAFKA_BROKERS
value: {{ .Values.kafka.brokerList }}
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
securityContext:
runAsUser: 0
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
volumeMounts:
- name: config
mountPath: /etc/metricbeat.yml
readOnly: true
subPath: metricbeat.yml
- name: modules
mountPath: /usr/share/metricbeat/modules.d
readOnly: true
volumes:
- name: config
configMap:
defaultMode: 0600
name: metricbeat-deployment-config
- name: modules
configMap:
defaultMode: 0600
name: metricbeat-deployment-modules

and my configmap

apiVersion: v1
kind: ConfigMap
metadata:
name: metricbeat-deployment-config
namespace: kube-system
labels:
k8s-app: metricbeat
data:
metricbeat.yml: |-
metricbeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
output.kafka:
hosts: ['${KAFKA_BROKERS}']
topic: {{ .Values.kafka.topic }}
required_acks: 1
compression: gzip
max_message_bytes: 1000000

apiVersion: v1
kind: ConfigMap
metadata:
name: metricbeat-deployment-modules
namespace: kube-system
labels:
k8s-app: metricbeat
data:
kubernetes.yml: |-
- module: kubernetes
enabled: true
metricsets:
- state_node
- state_deployment
- state_replicaset
- state_statefulset
- state_pod
- state_container
- event
period: 10s
host: ${NODE_NAME}
hosts: ["kube-state-metrics:8080"]
- module: kubernetes
enabled: true
metricsets:
- apiserver
#KUBERNETES_SERVICE_* env vars inherited automagically from cluster
hosts: ["https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}"]
#ssl.verification_mode: "none"
ssl.certificate_authorities: ["/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"]

I figured it out. I was missing the bearer_token for the apiserver

- module: kubernetes
  enabled: true
  metricsets:
    - apiserver
  #KUBERNETES_SERVICE_* env vars inherited automagically from cluster
  hosts: ["https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}"]
  #ssl.verification_mode: "none"
  ssl.certificate_authorities: ["/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"]
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.