Hello team,
I have Elastic Agent deployed with ECK, but for some reason I don't get the kubernetes metrics. Instead I see this error in the logs:
{"log.level":"info","@timestamp":"2023-10-10T14:55:42.666Z","message":"could not retrieve cluster metadata: fail to get kubernetes cluster metadata: unable to retrieve cluster identifiers","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"kubernetes/metrics-default","type":"kubernetes/metrics"},"log":{"source":"kubernetes/metrics-default"},"log.origin":{"file.line":740,"file.name":"util/kubernetes.go"},"service.name":"metricbeat","ecs.version":"1.6.0","ecs.version":"1.6.0"}
this is my elastic-agent-clusterrolebinding.yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: qa-apm-elastic-agent
subjects:
- kind: ServiceAccount
name: qa-apm-elastic-agent
namespace: qa-apm
roleRef:
kind: ClusterRole
name: qa-apm-elastic-agent
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
namespace: qa-apm
name: qa-apm-elastic-agent
subjects:
- kind: ServiceAccount
name: qa-apm-elastic-agent
namespace: qa-apm
roleRef:
kind: Role
name: qa-apm-elastic-agent
apiGroup: rbac.authorization.k8s.io
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: qa-apm-elastic-agent-kubeadm-config
namespace: qa-apm
subjects:
- kind: ServiceAccount
name: qa-apm-elastic-agent
namespace: qa-apm
roleRef:
kind: Role
name: qa-apm-elastic-agent-kubeadm-config
apiGroup: rbac.authorization.k8s.io
And this is my cluster role : cat elastic-agent-clusterrole.yaml
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: qa-apm-elastic-agent
labels:
k8s-app: qa-apm-elastic-agent
rules:
- apiGroups: [""] # "" indicates the core API group
resources:
- namespaces
- pods
- nodes
- nodes/metrics
- nodes/proxy
- nodes/stats
- events
- services
- configmaps
# Needed for cloudbeat
- serviceaccounts
- persistentvolumes
- persistentvolumeclaims
verbs:
- get
- watch
- list
- apiGroups: ["extensions"]
resources:
- replicasets
verbs:
- get
- list
- watch
- apiGroups: ["apps"]
resources:
- replicasets
- deployments
- replicasets
- daemonsets
verbs:
- get
- list
- watch
- apiGroups: ["batch"]
resources:
- jobs
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- nodes/stats
verbs:
- get
- nonResourceURLs:
- /metrics
verbs:
- get
- watch
- list
- nonResourceURLs:
- /metrics
verbs:
- get
- watch
- list
# Needed for cloudbeat
- apiGroups: ["rbac.authorization.k8s.io"]
resources:
- clusterrolebindings
- clusterroles
- rolebindings
- roles
verbs:
- get
- watch
- list
# Needed for cloudbeat
- apiGroups: ["policy"]
resources:
- podsecuritypolicies
verbs:
- get
- watch
- list
- apiGroups: ["coordination.k8s.io"]
resources:
- leases
verbs:
- get
- create
- update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: qa-apm-elastic-agent
# Should be the namespace where elastic-agent is running
namespace: qa-apm
labels:
k8s-app: qa-apm-elastic-agent
rules:
- apiGroups: ["coordination.k8s.io"]
resources:
- leases
verbs:
- get
- create
- update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: qa-apm-elastic-agent
namespace: qa-apm
labels:
k8s-app: qa-apm-elastic-agent
rules:
- apiGroups: [""]
resources:
- configmaps
resourceNames:
- kubeadm-config
verbs:
- get
Not sure if this is a permission issue. I can provide my kubernetes-integration file if required.