Unable to get the [Metricbeat Kubernetes] Overview Data in Kibana

Elastic Stack Elasticsearch
1

I have installed the Elasticsearch(8.16) and kibana(8.16) using RPMs in Centos vm.Also installed Metricbeats (8.16) in the centos vm using RPMS.

In the same vm i have kubernetes cluster and our application is running in the k3s cluster.I am trying to capture the kubernetes metrics and see that in Kibana dashboard.

Issue :
I am able to see the namesapce details in the dashboard but the pod, deployment fields i am getting as no data found. like this attached screenshot.

image
image1887×635 34.5 KB

this is the content of the In the

/etc/metricbeat/modules.d/kubernetes.yml

Node metrics, from kubelet:

  • module: kubernetes
    metricsets:
    • node
    • pod
    • system
    • volume
      period: 10s
      enabled: true
      hosts: ["https://127.0.0.1:6443"]
      bearer_token_file: /var/lib/metricbeat/kubernetes-app-token (This is the self signed token )
      ssl.verification_mode: "none"
      ssl.certificate_authorities: ["/var/lib/rancher/k3s/server/tls/client-ca.crt"]
      ssl.certificate: "/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt"
      ssl.key: "/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key"
      add_metadata: true
      kubeconfig: "/etc/rancher/k3s/k3s.yaml"
      node: node2
      namespace: default

Below is the error message i am getting in the logs of Metricbeat :

{"log.level":"error","@timestamp":"2024-12-03T11:11:26.621Z","log.logger":"kubernetes.node","log.origin":{"function":"github.com/elastic/beats/v7/metricbeat/module/kubernetes/node.(*MetricSet).Fetch","file.name":"node/node.go","file.line":91},"message":"HTTP error 404 in : 404 Not Found","service.name":"metricbeat","ecs.version":"1.6.0"}

{"log.level":"error","@timestamp":"2024-12-03T11:11:26.621Z","log.origin":{"function":"github.com/elastic/beats/v7/metricbeat/mb/module.(*metricSetWrapper).fetch","file.name":"module/wrapper.go","file.line":266},"message":"Error fetching data for metricset kubernetes.system: error doing HTTP request to fetch 'system' Metricset data: HTTP error 404 in : 404 Not Found","service.name":"metricbeat","ecs.version":"1.6.0"}

{"log.level":"error","@timestamp":"2024-12-03T11:11:26.621Z","log.logger":"kubernetes.pod","log.origin":{"function":"github.com/elastic/beats/v7/metricbeat/module/kubernetes/pod.(*MetricSet).Fetch","file.name":"pod/pod.go","file.line":92},"message":"HTTP error 404 in : 404 Not Found","service.name":"metricbeat","ecs.version":"1.6.0"}

{"log.level":"error","@timestamp":"2024-12-03T11:11:26.621Z","log.origin":{"function":"github.com/elastic/beats/v7/metricbeat/mb/module.(*metricSetWrapper).fetch","file.name":"module/wrapper.go","file.line":266},"message":"Error fetching data for metricset kubernetes.volume: error doing HTTP request to fetch 'volume' Metricset data: HTTP error 404 in : 404 Not Found","service.name":"metricbeat","ecs.version":"1.6.0"}

Note : The Metricbeats is installed as RPM and not inside kubernetes cluster.But both are in same vm.

Please guide us fixing this issue , Thanks in advance.