Metricbeat in Kubernetes - 404 error for Kubernetes module events

Elastic Stack Beats
1

Hello,

I am running Metricbeat 7.0.1 on Kubernetes cluster (Server V1.14), followed the official instructions, and enabled System, Docker, Kubernetes modules. System and Docker modules working where I can see the data in Kibana (Discover and Dashboards). But for all Kubernetes events I am getting the following error.message
HTTP error 404 in : 404 Not Found

Following is my Kubernetes module config

- module: kubernetes
  period: 10s
  enabled: true
  metricsets:
    - container
    - node
    - pod
    - system
    - volume
  host: ${NODE_NAME}
  hosts: ["https://${KUBE_HOST}"]
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  ssl.certificate_authorities:
    - /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

where KUBE_HOST is 192.168.200.75:6443 which I got this info from executing the following command

>kubectl cluster-info
Kubernetes master is running at https://192.168.200.75:6443

Serviceaccount config

---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: metricbeat
subjects:
- kind: ServiceAccount
  name: metricbeat
  namespace: dev
roleRef:
  kind: ClusterRole
  name: metricbeat
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: metricbeat
  labels:
    app: metricbeat
rules:
- apiGroups: [""]
  resources:
  - nodes
  - namespaces
  - events
  - pods  
  verbs: ["get", "list", "watch"]
- apiGroups: ["extensions"]
  resources:
  - replicasets
  verbs: ["get", "list", "watch"]
- apiGroups: ["apps"]
  resources:
  - statefulsets
  - deployments
  verbs: ["get", "list", "watch"]
- apiGroups:
  - ""
  resources:
  - nodes/stats
  - nodes/metrics
  verbs:
  - get
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: metricbeat
  namespace: dev
  labels:
    app: metricbeat
---

Initially I was getting HTTP error 403 in : 403 Forbidden error in the events, then I changed the ClusterroleBinding.roleref.name to "cluster-admin" temporarily so that metricbeat will access Kubernetes with cluster-admin role, now I am getting 404 error.

Do I need to configure anything in metricbeat or Kubernetes to get the metrics?

Thanks

2

@chandra2037 Sorry for the late response here. So you are trying to use k8s module in metricbeat to get metrics from the k8s cluster that metricbeat is running on? My first guess is hosts: ["https://${KUBE_HOST}"] should be pointing to hosts: ["localhost:10255"]. Also maybe Kubernetes data collection error with metricbeat! will help!

3

I am still looking for a solution, so thank you for responding @Kaiyan_Sheng. I will look into the discussion that you provided.

4

Finally able to get it working.
Followed the instructions given by this post
Where configured hosts as https://${NODE_NAME}:10250, then also I was getting “x509: certificate signed by unknown authority” error. As per this post disabled SSL verification for now, then able to get the Kubernetes metrics.

2 Likes
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.