Issue with kubernetes metric data fetch by meteicbeat

[Sam96]

I have installed the Elasticsearch(8.16) and kibana(8.16) using RPMs in Centos vm.Also installed Metricbeats (8.16) in the centos vm using RPMS.

In the same vm i have kubernetes cluster and our application is running in the k3s cluster.I am trying to capture the kubernetes metrics and see that in Kibana dashboard.

Issue :
I am able to see the namesapce details in the dashboard but the pod, deployment fields i am getting as no data found. like this attached screenshot.

this is the content of the In the

/etc/metricbeat/modules.d/kubernetes.yml

Node metrics, from kubelet:

• module: kubernetes
  metricsets:
  • node
  • pod
  • system
  • volume
    period: 10s
    enabled: true
    hosts: ["https://127.0.0.1:6443"]**
    bearer_token_file: /var/lib/metricbeat/kubernetes-app-token (This is the self signed token )
    ssl.verification_mode: "none"
    ssl.certificate_authorities: ["/var/lib/rancher/k3s/server/tls/client-ca.crt"]
    ssl.certificate: "/var/lib/rancher/k3s/server/tls/client-kube-apiserver.crt"
    ssl.key: "/var/lib/rancher/k3s/server/tls/client-kube-apiserver.key"
    add_metadata: true
    kubeconfig: "/etc/rancher/k3s/k3s.yaml"
    node: node2
    namespace: default

Below is the error message i am getting in the logs of Metricbeat :

{"log.level":"error","@timestamp":"2024-12-03T11:11:26.621Z","log.logger":"kubernetes.node","log.origin":{"function":"github.com/elastic/beats/v7/metricbeat/module/kubernetes/node.(*MetricSet).Fetch","file.name":"node/node.go","file.line":91},"message":"HTTP error 404 in : 404 Not Found","service.name":"metricbeat","ecs.version":"1.6.0"}

{"log.level":"error","@timestamp":"2024-12-03T11:11:26.621Z","log.origin":{"function":"github.com/elastic/beats/v7/metricbeat/mb/module.(*metricSetWrapper).fetch","file.name":"module/wrapper.go","file.line":266},"message":"Error fetching data for metricset kubernetes.system: error doing HTTP request to fetch 'system' Metricset data: HTTP error 404 in : 404 Not Found","service.name":"metricbeat","ecs.version":"1.6.0"}

{"log.level":"error","@timestamp":"2024-12-03T11:11:26.621Z","log.logger":"kubernetes.pod","log.origin":{"function":"github.com/elastic/beats/v7/metricbeat/module/kubernetes/pod.(*MetricSet).Fetch","file.name":"pod/pod.go","file.line":92},"message":"HTTP error 404 in : 404 Not Found","service.name":"metricbeat","ecs.version":"1.6.0"}

{"log.level":"error","@timestamp":"2024-12-03T11:11:26.621Z","log.origin":{"function":"github.com/elastic/beats/v7/metricbeat/mb/module.(*metricSetWrapper).fetch","file.name":"module/wrapper.go","file.line":266},"message":"Error fetching data for metricset kubernetes.volume: error doing HTTP request to fetch 'volume' Metricset data: HTTP error 404 in : 404 Not Found","service.name":"metricbeat","ecs.version":"1.6.0"}

Note : The Metricbeats is installed as RPM and not inside kubernetes cluster.But both are in same vm.

Please guide us fixing this issue , Thanks in advance.

[stephenb (Stephen Brown)]

@Sam96 Please open only 1 topic and give it a bit of time. You are a new user, so the anti-spam filters will check because your post contains links.

I will close the other topic.

[stephenb (Stephen Brown)]

If you want to monitor Inside the kubernetes you should follow this in my opinion....

Run Metricbeat on Kubernetes | Metricbeat Reference [8.16] | Elastic

If you want to monitor from outside you will need to follow this in detail

Kubernetes module | Metricbeat Reference [8.16] | Elastic

The error message indicate that those dataset are not metricsset are not available at the address:port you are providing...

Perhaps install metricbeat into the cluster....

[Sam96]

Hello ,

@stephenb , Thanks for your response .

For the first approach

we tried installing using the mainfest file Inside the kubernetes ,but seems like we are unable to fetch the metricbeats image and it is giving error
containers:
- name: metricbeat
image: docker.elastic.co/beats/metricbeat-wolfi:8.16.1

Errror : Error response from daemon: Get "https://docker.elastic.co/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

For the second approach

if we are running it outside the kubernetes cluster what should be the ideal value for the address:port ,
and how to verify that those metricsset are available or not at the address:port we are providing

[stephenb (Stephen Brown)]

I just tried the sample metricbeat deployment file with only replacing the credentials and I could deploy to my K8s

Perhaps you can not pull from the public docker Firewall or something... you would need to determine why. Here simple docker pull as well confirms good.

$ docker pull docker.elastic.co/beats/metricbeat-wolfi:8.16.1
8.16.1: Pulling from beats/metricbeat-wolfi
Digest: sha256:430a516d882e1ea1d8edc6a8e044b7cb384cf08405f3ab985f3de96e66e4199d
Status: Image is up to date for docker.elastic.co/beats/metricbeat-wolfi:8.16.1
docker.elastic.co/beats/metricbeat-wolfi:8.16.1

For deploying outside \You can also look at this

beats/deploy/kubernetes/metricbeat at main · elastic/beats

You will need to inspect your cluster to see what the ports are on perhaps using something like lens

Lens | The Kubernetes IDE