Elastic-agent kubernetes-integration does not collect kubernetes metrics

[tara]

Hello team,

I have Elastic Agent deployed with ECK, but for some reason I don't get the kubernetes metrics. Instead I see this error in the logs:

{"log.level":"info","@timestamp":"2023-10-10T14:55:42.666Z","message":"could not retrieve cluster metadata: fail to get kubernetes cluster metadata: unable to retrieve cluster identifiers","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"kubernetes/metrics-default","type":"kubernetes/metrics"},"log":{"source":"kubernetes/metrics-default"},"log.origin":{"file.line":740,"file.name":"util/kubernetes.go"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}

this is my elastic-agent-clusterrolebinding.yaml

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: qa-apm-elastic-agent
subjects:
- kind: ServiceAccount
  name: qa-apm-elastic-agent
  namespace: qa-apm
roleRef:
  kind: ClusterRole
  name: qa-apm-elastic-agent
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: qa-apm
  name: qa-apm-elastic-agent
subjects:
  - kind: ServiceAccount
    name: qa-apm-elastic-agent
    namespace: qa-apm
roleRef:
  kind: Role
  name: qa-apm-elastic-agent
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: qa-apm-elastic-agent-kubeadm-config
  namespace: qa-apm
subjects:
  - kind: ServiceAccount
    name: qa-apm-elastic-agent
    namespace: qa-apm
roleRef:
  kind: Role
  name: qa-apm-elastic-agent-kubeadm-config
  apiGroup: rbac.authorization.k8s.io

And this is my cluster role : cat elastic-agent-clusterrole.yaml
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: qa-apm-elastic-agent
  labels:
    k8s-app: qa-apm-elastic-agent
rules:
- apiGroups: [""] # "" indicates the core API group
  resources:
  - namespaces
  - pods
  - nodes
  - nodes/metrics
  - nodes/proxy
  - nodes/stats
  - events
  - services
  - configmaps
  # Needed for cloudbeat
  - serviceaccounts
  - persistentvolumes
  - persistentvolumeclaims
  verbs:
  - get
  - watch
  - list
- apiGroups: ["extensions"]
  resources:
  - replicasets
  verbs:
  - get
  - list
  - watch
- apiGroups: ["apps"]
  resources:
  - replicasets
  - deployments
  - replicasets
  - daemonsets
  verbs:
  - get
  - list
  - watch
- apiGroups: ["batch"]
  resources:
  - jobs
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - ""
  resources:
  - nodes/stats
  verbs:
  - get
- nonResourceURLs:
  - /metrics
  verbs:
  - get
  - watch
  - list
- nonResourceURLs:
  - /metrics
  verbs:
  - get
  - watch
  - list
# Needed for cloudbeat
- apiGroups: ["rbac.authorization.k8s.io"]
  resources:
  - clusterrolebindings
  - clusterroles
  - rolebindings
  - roles
  verbs:
  - get
  - watch
  - list
# Needed for cloudbeat
- apiGroups: ["policy"]
  resources:
  - podsecuritypolicies
  verbs:
  - get
  - watch
  - list
- apiGroups: ["coordination.k8s.io"]
  resources:
  - leases
  verbs:
  - get
  - create
  - update

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: qa-apm-elastic-agent
  # Should be the namespace where elastic-agent is running
  namespace: qa-apm
  labels:
    k8s-app: qa-apm-elastic-agent
rules:
- apiGroups: ["coordination.k8s.io"]
  resources:
  - leases
  verbs:
  - get
  - create
  - update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: qa-apm-elastic-agent
  namespace: qa-apm
  labels:
    k8s-app: qa-apm-elastic-agent
rules:
- apiGroups: [""]
  resources:
  - configmaps
  resourceNames:
  - kubeadm-config
  verbs:
  - get

Not sure if this is a permission issue. I can provide my kubernetes-integration file if required.