Metricbeat in Kubernetes - 404 error for Kubernetes module events

(Chandra) #1

Hello,

I am running Metricbeat 7.0.1 on Kubernetes cluster (Server V1.14), followed the official instructions, and enabled System, Docker, Kubernetes modules. System and Docker modules working where I can see the data in Kibana (Discover and Dashboards). But for all Kubernetes events I am getting the following error.message
HTTP error 404 in : 404 Not Found

Following is my Kubernetes module config

- module: kubernetes
  period: 10s
  enabled: true
  metricsets:
    - container
    - node
    - pod
    - system
    - volume
  host: ${NODE_NAME}
  hosts: ["https://${KUBE_HOST}"]
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  ssl.certificate_authorities:
    - /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

where KUBE_HOST is 192.168.200.75:6443 which I got this info from executing the following command

>kubectl cluster-info
Kubernetes master is running at https://192.168.200.75:6443

Serviceaccount config

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: metricbeat
subjects:
- kind: ServiceAccount
  name: metricbeat
  namespace: dev
roleRef:
  kind: ClusterRole
  name: metricbeat
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: metricbeat
  labels:
    app: metricbeat
rules:
- apiGroups: [""]
  resources:
  - nodes
  - namespaces
  - events
  - pods  
  verbs: ["get", "list", "watch"]
- apiGroups: ["extensions"]
  resources:
  - replicasets
  verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
  resources:
  - statefulsets
  - deployments
  verbs: ["get", "list", "watch"]
- apiGroups:
  - ""
  resources:
  - nodes/stats
  - nodes/metrics
  verbs:
  - get
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: metricbeat
  namespace: dev
  labels:
    app: metricbeat
---

Initially I was getting HTTP error 403 in : 403 Forbidden error in the events, then I changed the ClusterroleBinding.roleref.name to "cluster-admin" temporarily so that metricbeat will access Kubernetes with cluster-admin role, now I am getting 404 error.

Do I need to configure anything in metricbeat or Kubernetes to get the metrics?

Thanks