Shiping logs from kubernetes to elasticsearch using beats - get: "clusterroles.rbac.authorization.k8s.io "filebeat" is forbidden: attempt to grant extra privileges"

Elastic Stack Beats
1

following
https://www.elastic.co/guide/en/beats/filebeat/6.1/running-on-kubernetes.html

and the following post:

curl -L -O https://raw.githubusercontent.com/elastic/beats/6.0/deploy/kubernetes/filebeat-kubernetes.yaml

here is full yaml:


I edited variables to point to my instance:

           - name: ELASTICSEARCH_HOST
          value: xxxxxxxxxx.us-central1.gcp.cloud.es.io
        - name: ELASTICSEARCH_PORT
          value: "9243"
        - name: ELASTICSEARCH_USERNAME
          value: elastic
        - name: ELASTICSEARCH_PASSWORD
          value: xxxxxxx

I ran kubectl apply -f filebeat-kubernetes.yaml

and get the following error:

    configmap "filebeat-config" created
configmap "filebeat-prospectors" created
daemonset "filebeat" created
clusterrolebinding "filebeat" created
serviceaccount "filebeat" created
Error from server (Forbidden): error when creating "filebeat-kubernetes.yaml": clusterroles.rbac.authorization.k8s.io "filebeat" is forbidden: attempt to grant extra privileges: [PolicyRule{Resources:["namespaces"], APIGroups:[""], Verbs:["get"]} PolicyRule{Resources:["namespaces"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["namespaces"], APIGroups:[""], Verbs:["list"]} PolicyRule{Resources:["pods"], APIGroups:[""], Verbs:["get"]} PolicyRule{Resources:["pods"], APIGroups:[""], Verbs:["watch"]} PolicyRule{Resources:["pods"], APIGroups:[""], Verbs:["list"]}] user=&{bob@example.com  [system:authenticated] map[]} ownerrules=[PolicyRule{Resources:["selfsubjectaccessreviews"], APIGroups:["authorization.k8s.io"], Verbs:["create"]} PolicyRule{NonResourceURLs:["/api" "/api/*" "/apis" "/apis/*" "/healthz" "/swaggerapi" "/swaggerapi/*" "/version"], Verbs:["get"]}] ruleResolutionErrors=[]

beautify error:

Error from server (Forbidden): error when creating "filebeat-kubernetes.yaml": clusterroles.rbac.authorization.k8s.io "filebeat" is forbidden: attempt to grant extra privileges: 
    [
PolicyRule{Resources:["namespaces"], APIGroups:[""], Verbs:["get"]}
PolicyRule{Resources:["namespaces"], APIGroups:[""], Verbs:["watch"]}
PolicyRule{Resources:["namespaces"], APIGroups:[""], Verbs:["list"]}
PolicyRule{Resources:["pods"], APIGroups:[""], Verbs:["get"]}
PolicyRule{Resources:["pods"], APIGroups:[""], Verbs:["watch"]}
PolicyRule{Resources:["pods"], APIGroups:[""], Verbs:["list"]}]
user=&{bob@example.com  [system:authenticated] map[]}
ownerrules=[
  PolicyRule{
    Resources:["selfsubjectaccessreviews"],
    APIGroups:["authorization.k8s.io"],
    Verbs:["create"]}
  PolicyRule{NonResourceURLs:["/api" "/api/*" "/apis" "/apis/*" "/healthz" "/swaggerapi" "/swaggerapi/*" "/version"],
              Verbs:["get"]}]
ruleResolutionErrors=[]
2

Hi @dinaWork,

Depending on your cluster settings this error may not be fatal. But let me explain what's going on:

We setup RBAC roles to ensure filebeat has access to pods metadata when enriching logs.

While this is what you want, it requires you have the cluster-admin role when deploying it, you can bind this role to your user doing something like this:

kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin --user=carlos@elastic.co

Just replace the user name with yours.

3

hi @exekias, thanks for your reply, binding the role solved my issue!!

now beats fails
get the following log:

ERR Failed to connect: Get http://xxxxxx.us-central1.gcp.cloud.es.io:9243: read tcp 13.17.4.10:40766->31.177.77.2:9243: read: connection reset by peer
11:38:19.230  reload.go:222: INFO Dynamic config reloader stopped
11:38:19.230  crawler.go:135: INFO Crawler stopped
11:38:19.230  registrar.go:210: INFO Stopping Registrar
11:38:19.230  registrar.go:165: INFO Ending Registrar
11:38:19.231  forwarder.go:35: INFO Prospector outlet closed
11:38:19.234  metrics.go:51: INFO Total non-zero values: beat.info.uptime.ms=40449 beat.memstats.gc_next=25194960 beat.memstats.memory_alloc=13894472 beat.memstats.memory_total=43826552 filebeat.events.active=4130 filebeat.events.added=4140 filebeat.events.done=10 filebeat.harvester.closed=4 filebeat.harvester.open_files=16 filebeat.harvester.running=16 filebeat.harvester.started=20 libbeat.config.module.running=1 libbeat.config.module.starts=1 libbeat.config.reloads=2 libbeat.output.read.errors=6 libbeat.output.type=elasticsearch libbeat.output.write.bytes=1434 libbeat.pipeline.clients=0 libbeat.pipeline.events.active=4116 libbeat.pipeline.events.failed=4 libbeat.pipeline.events.filtered=20 libbeat.pipeline.events.published=4116 libbeat.pipeline.events.retry=150 libbeat.pipeline.events.total=4140 registrar.states.current=6 registrar.states.update=6 registrar.writes=8
11:38:19.234  metrics.go:52: INFO Uptime: 40.449909073s
11:38:19.234  beat.go:284: INFO filebeat stopped.
4

It looks like you are trying to use http instead of https, try changing the host value