I've been trying to setup metricbeat to monitor my k8s cluster. I've got my daemonset and pods working, but I'm having difficulty getting apiserver metrics. Originally I was getting ssl cert errors from metricbeat until I set the "ssl.certificate_authorities" option in my kubernetes module. Unfortunately I am now receiving nothing from the apiserver and no errors are showing up in the log. I suspect this is an RBAC issue because if I attempt to curl the /metrics endpoint I get an error.
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {},
"status": "Failure",
"message": "forbidden: User "system:anonymous" cannot get path "/metrics"",
"reason": "Forbidden",
"details": {},
"code": 403
}
I have configured my daemonset and pod to use the metricbeat service account and I think that I've granted the appropriate permissions, however I must be missing something.
Here is the role config I'm binding to the service account
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: metricbeat
labels:
k8s-app: metricbeat
rules:
- apiGroups: [""]
resources:
- nodes
- namespaces
- events
- pods
verbs: ["get", "list", "watch"]- apiGroups: ["extensions"]
resources:
- replicasets
verbs: ["get", "list", "watch"]- apiGroups: ["apps"]
resources:
- statefulsets
- deployments
verbs: ["get", "list", "watch"]- apiGroups:
- ""
resources:- nodes/stats
- nodes/metrics
verbs:- get
- nonResourceURLs:
- "/metrics"
verbs:- get
Here is my pod deployment
Deploy singleton instance in the whole cluster for some unique data sources, like kube-state-metrics
apiVersion: apps/v1beta1
kind: Deployment
metadata:
name: metricbeat
namespace: kube-system
labels:
k8s-app: metricbeat
spec:
template:
metadata:
labels:
k8s-app: metricbeat
spec:
serviceAccountName: metricbeat
hostNetwork: true
dnsPolicy: ClusterFirstWithHostNet
containers:
- name: metricbeat
image: {{ .Values.beatImage }}
args: [
"-c", "/etc/metricbeat.yml",
"-e",
]
resources:
limits:
cpu: 200m
memory: 100Mi
requests:
cpu: 50m
memory: 50Mi
env:
- name: KAFKA_BROKERS
value: {{ .Values.kafka.brokerList }}
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
securityContext:
runAsUser: 0
resources:
limits:
memory: 200Mi
requests:
cpu: 100m
memory: 100Mi
volumeMounts:
- name: config
mountPath: /etc/metricbeat.yml
readOnly: true
subPath: metricbeat.yml
- name: modules
mountPath: /usr/share/metricbeat/modules.d
readOnly: true
volumes:
- name: config
configMap:
defaultMode: 0600
name: metricbeat-deployment-config
- name: modules
configMap:
defaultMode: 0600
name: metricbeat-deployment-modules
and my configmap
apiVersion: v1
kind: ConfigMap
metadata:
name: metricbeat-deployment-config
namespace: kube-system
labels:
k8s-app: metricbeat
data:
metricbeat.yml: |-
metricbeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
output.kafka:
hosts: ['${KAFKA_BROKERS}']
topic: {{ .Values.kafka.topic }}
required_acks: 1
compression: gzip
max_message_bytes: 1000000apiVersion: v1
kind: ConfigMap
metadata:
name: metricbeat-deployment-modules
namespace: kube-system
labels:
k8s-app: metricbeat
data:
kubernetes.yml: |-
- module: kubernetes
enabled: true
metricsets:
- state_node
- state_deployment
- state_replicaset
- state_statefulset
- state_pod
- state_container
- event
period: 10s
host: ${NODE_NAME}
hosts: ["kube-state-metrics:8080"]
- module: kubernetes
enabled: true
metricsets:
- apiserver
#KUBERNETES_SERVICE_* env vars inherited automagically from cluster
hosts: ["https://${KUBERNETES_SERVICE_HOST}:${KUBERNETES_SERVICE_PORT}"]
#ssl.verification_mode: "none"
ssl.certificate_authorities: ["/var/run/secrets/kubernetes.io/serviceaccount/ca.crt"]