Kibana/ES not showing kube pod metrics

Elastic Stack Beats
,
1

I have ES-Beats-Kibana setup running in Kubernetes. In Kibana under Infrastructure App on Host Tab I can see all metrics (CPU, Memory, Load, In/Out Traffic & Log Rate) for all Kubernetes nodes.

But On Kubernetes Tab, I see list of all containers running in the cluster, but don't see any metric (CPU, Memory, In/Out Traffic). All these metrics are showing 0%

I do have metricbeat & filebeat running as Daemonset in the cluster. Don't know if it doesn't collect these pod metrics or Kibana doesn't show it.

14%20AM
Screen Shot 2019-11-21 at 9.52.14 AM.png1224×928 56.7 KB

This is manifest for metricbeat

Any idea whats wrong here ?

2

Hi @rp346,

+1, I have the same issue.

If I use prometheus + grafana, i can see pods metrics.

So, the problem seem be due to metricbeats ...

The serviceAccount can't list some ressources:

Failed to list *v1beta1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:kube-system:metricbeat" cannot list resource "replicasets" in API group "apps" at the cluster scope

An issue is created here

The problems seem be fix in future release

3

To solve the problem with serviceAccount, edit the serviceaccountname metricbeat:

k -n kube-system edit clusterroles metricbeat

And change some elements:

[...]
rules:
- apiGroups:
  - ""
  - extensions
  - apps
  resources:
  - namespaces
  - pods
  - events
  - deployments
  - nodes
  - replicasets
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - extensions
  resources:
  - replicasets
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - apps
  resources:
  - statefulsets
  - deployments
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes/stats
  verbs:
  - get
[...]
4

I updated the metricbeat ClusterRole as per your suggestion and re-deployed metricbeat. Still I don't see any metric under Kubernetes tab in Infrastructure App.

Yes Prometheus with Grafana works perfectly which has node-exporter.

5

Yes, I'am exactly in the same case of you.

Update the clusterRole don't fix the problem

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.