Kibana/ES not showing kube pod metrics

[rp346]

I have ES-Beats-Kibana setup running in Kubernetes. In Kibana under Infrastructure App on Host Tab I can see all metrics (CPU, Memory, Load, In/Out Traffic & Log Rate) for all Kubernetes nodes.

But On Kubernetes Tab, I see list of all containers running in the cluster, but don't see any metric (CPU, Memory, In/Out Traffic). All these metrics are showing 0%

I do have metricbeat & filebeat running as Daemonset in the cluster. Don't know if it doesn't collect these pod metrics or Kibana doesn't show it.

This is manifest for metricbeat

Any idea whats wrong here ?

[Benjamin_Carriou]

Hi @rp346,

+1, I have the same issue.

If I use prometheus + grafana, i can see pods metrics.

So, the problem seem be due to metricbeats ...

The serviceAccount can't list some ressources:

Failed to list *v1beta1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:kube-system:metricbeat" cannot list resource "replicasets" in API group "apps" at the cluster scope

An issue is created here

The problems seem be fix in future release

[Benjamin_Carriou]

To solve the problem with serviceAccount, edit the serviceaccountname metricbeat:

k -n kube-system edit clusterroles metricbeat

And change some elements:

[...]
rules:
- apiGroups:
  - ""
  - extensions
  - apps
  resources:
  - namespaces
  - pods
  - events
  - deployments
  - nodes
  - replicasets
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - extensions
  resources:
  - replicasets
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - apps
  resources:
  - statefulsets
  - deployments
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes/stats
  verbs:
  - get
[...]

[rp346]

I updated the metricbeat ClusterRole as per your suggestion and re-deployed metricbeat. Still I don't see any metric under Kubernetes tab in Infrastructure App.

Yes Prometheus with Grafana works perfectly which has node-exporter.

[Benjamin_Carriou]

Yes, I'am exactly in the same case of you.

Update the clusterRole don't fix the problem