Failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:es-common:elastic-agent"

Elastic Observability
1

Hello,
I have installed ES, Kibana 8.9.1 on EKS 1.27 and struggling with fleet.

I get following error messages fleet-server-agent pod logs:

{"log.level":"error","@timestamp":"2023-09-06T12:16:07.511Z","message":"E0906 12:16:07.511011    1111 reflector.go:138] k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User \"system:serviceaccount:es-common:elastic-agent\" cannot list resource \"replicasets\" in API group \"apps\" at the cluster scope","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-09-06T12:16:13.920Z","message":"W0906 12:16:13.920011    1079 reflector.go:324] k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User \"system:serviceaccount:es-common:elastic-agent\" cannot list resource \"replicasets\" in API group \"apps\" at the cluster scope","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-09-06T12:16:13.920Z","message":"E0906 12:16:13.920071    1079 reflector.go:138] k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User \"system:serviceaccount:es-common:elastic-agent\" cannot list resource \"replicasets\" in API group \"apps\" at the cluster scope","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"ecs.version":"1.6.0"}
W0906 12:16:23.490412     996 reflector.go:324] k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:es-common:elastic-agent" cannot list resource "replicasets" in API group "apps" at the cluster scope
E0906 12:16:23.490459     996 reflector.go:138] k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:es-common:elastic-agent" cannot list resource "replicasets" in API group "apps" at the cluster scope
{"log.level":"error","@timestamp":"2023-09-06T12:16:25.044Z","message":"W0906 12:16:25.043438    1070 reflector.go:324] k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User \"system:serviceaccount:es-common:elastic-agent\" cannot list resource \"replicasets\" in API group \"apps\" at the cluster scope","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-09-06T12:16:25.045Z","message":"E0906 12:16:25.043477    1070 reflector.go:138] k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User \"system:serviceaccount:es-common:elastic-agent\" cannot list resource \"replicasets\" in API group \"apps\" at the cluster scope","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-09-06T12:16:32.413Z","message":"W0906 12:16:32.413141    1091 reflector.go:324] k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User \"system:serviceaccount:es-common:elastic-agent\" cannot list resource \"replicasets\" in API group \"apps\" at the cluster scope","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"log-default","type":"log"},"log":{"source":"log-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-09-06T12:16:32.413Z","message":"E0906 12:16:32.413178    1091 reflector.go:138] k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User \"system:serviceaccount:es-common:elastic-agent\" cannot list resource \"replicasets\" in API group \"apps\" at the cluster scope","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"log-default","type":"log"},"log":{"source":"log-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-09-06T12:16:45.246Z","message":"W0906 12:16:45.245961    1078 reflector.go:324] k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User \"system:serviceaccount:es-common:elastic-agent\" cannot list resource \"replicasets\" in API group \"apps\" at the cluster scope","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-09-06T12:16:45.246Z","message":"E0906 12:16:45.246003    1078 reflector.go:138] k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User \"system:serviceaccount:es-common:elastic-agent\" cannot list resource \"replicasets\" in API group \"apps\" at the cluster scope","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"filestream-monitoring","type":"filestream"},"log":{"source":"filestream-monitoring"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-09-06T12:16:49.793Z","message":"W0906 12:16:49.793893    1111 reflector.go:324] k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User \"system:serviceaccount:es-common:elastic-agent\" cannot list resource \"replicasets\" in API group \"apps\" at the cluster scope","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-09-06T12:16:49.794Z","message":"E0906 12:16:49.793926    1111 reflector.go:138] k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User \"system:serviceaccount:es-common:elastic-agent\" cannot list resource \"replicasets\" in API group \"apps\" at the cluster scope","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"system/metrics-default","type":"system/metrics"},"log":{"source":"system/metrics-default"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-09-06T12:16:59.204Z","message":"W0906 12:16:59.203781    1079 reflector.go:324] k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User \"system:serviceaccount:es-common:elastic-agent\" cannot list resource \"replicasets\" in API group \"apps\" at the cluster scope","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-09-06T12:16:59.204Z","message":"E0906 12:16:59.203989    1079 reflector.go:138] k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User \"system:serviceaccount:es-common:elastic-agent\" cannot list resource \"replicasets\" in API group \"apps\" at the cluster scope","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"beat/metrics-monitoring","type":"beat/metrics"},"log":{"source":"beat/metrics-monitoring"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-09-06T12:17:12.300Z","message":"W0906 12:17:12.300171    1070 reflector.go:324] k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User \"system:serviceaccount:es-common:elastic-agent\" cannot list resource \"replicasets\" in API group \"apps\" at the cluster scope","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-09-06T12:17:12.300Z","message":"E0906 12:17:12.300202    1070 reflector.go:138] k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User \"system:serviceaccount:es-common:elastic-agent\" cannot list resource \"replicasets\" in API group \"apps\" at the cluster scope","component":{"binary":"metricbeat","dataset":"elastic_agent.metricbeat","id":"http/metrics-monitoring","type":"http/metrics"},"log":{"source":"http/metrics-monitoring"},"ecs.version":"1.6.0"}
W0906 12:17:19.524046     996 reflector.go:324] k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:es-common:elastic-agent" cannot list resource "replicasets" in API group "apps" at the cluster scope
E0906 12:17:19.524085     996 reflector.go:138] k8s.io/client-go@v0.23.4/tools/cache/reflector.go:167: Failed to watch *v1.ReplicaSet: failed to list *v1.ReplicaSet: replicasets.apps is forbidden: User "system:serviceaccount:es-common:elastic-agent" cannot list resource "replicasets" in API group "apps" at the cluster scope

My clusterrole definition is the default one which is:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: elastic-agent
rules:
- apiGroups: [""] # "" indicates the core API group
  resources:
  - pods
  - nodes
  - namespaces
  verbs:
  - get
  - watch
  - list
- apiGroups: ["coordination.k8s.io"]
  resources:
  - leases
  verbs:
  - get
  - create
  - update

Although I add "-replicasets" under apiGroups section, nothing changes.

Could you please advise what I am missing?

Thanks & Regards

1 Like
2

Hi,

replicasets belongs to apps apiGroup.

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: elastic-agent
rules:
- apiGroups: ["","apps"]