Filebeat running on ECK cannot create an index

[cloudmatt (Matt)]

I'm trying to wrap my head around running ECK and using filebeat. I've successfully deployed filebeat and I see it grabbing logs from my GCP pub/sub topic but it keeps dropping them because of the following error:

Private:(*pubsub.Message)(0xc00047e9a0), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:mapstr.M(nil)}} (status=403): {\"type\":\"security_exception\",\"reason\":\"action [indices:admin/auto_create] is unauthorized for user [elastic-filebeat-beat-user] with roles [beats_admin,eck_beat_es_filebeat_role_v77,ingest_admin,kibana_admin,remote_monitoring_agent] on indices [google-auditlog-2022.10.12], this action is granted by the index privileges [auto_configure,create_index,manage,all]\"}, dropping event!","service.name":"filebeat","ecs.version":"1.6.0"}

it's saying filbeat doesn't have the permissions to auto create an index but I don't see anywhere where I can change these permissions or update filebeat to use some other role. Could I get some guidance on how to resolve? Relevant snippet from k8s manifest:

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: filebeat
spec:
  type: filebeat
  version: 8.4.3
  elasticsearchRef:
    name: elasticsearch
  kibanaRef:
    name: kibana
  config:
    filebeat.inputs:
    - type: gcp-pubsub
      project_id: mycoolproject
      topic: topic-topic-topic
      subscription.name: topic-sub-sub
      tags: ["google", "google-workspace-auditlog"]
      json.fields_under_root: true
      index: "google-auditlog-%{+yyyy.MM.dd}"

I've followed the 'getting started' guide and have the ECK operator and all the CRDs installed as well. The logs from the operator statefulset don't show any errors or anything either.

[krische]

I think I'm running into the same issue as you. I want one of my filebeat instances to send data to a different Elasticsearch index. But when I do that, I see the same errors as you.

From what I can read in the ECK documentation, it sounds like ECK is automatically creating a user in Elasticsearch for each beat instance. My guess is that the user generated is probably only given privileges for the default index name, like filebeat-*. I can't find any proof of that though or if it's even possible to change that if that is the case.

[cloudmatt (Matt)]

Yeah - I tried looking through all the docs and couldn't make sense of it. Unfortunate there hasn't been a response to this post

Guess i'll just shove everything into a single index and hope it works for the best

[khaled_belgacem (khaled)]

Hello Matt,

i faced the same probleme 2 days ago and read your topic while looking for a solution, now i just solved it and i thought you may still looking for a way to do this, here's how it worked for me:

in the yaml file for your filebeat under output.elasticsearch and setup.kibana specify the username and the password of an elasticsearch user that you have to create manualy from kibana; give that user the appropriate privillages over the index you need to create by adding a new role, please find here below my config file and a screenshot for a clear understanding

apiVersion: beat.k8s.elastic.co/v1beta1
kind: Beat
metadata:
  name: filebeatvpn-{{.Values.Kubernetes.Namespace}}
  namespace: {{.Values.Kubernetes.Namespace}}
spec:
  type: filebeat
  version: {{.Values.Elastic.STACK_VERSION}}
  elasticsearchRef:
    name: {{.Values.Elastic.ECN}}
  kibanaRef:
    name: {{.Values.Kibana.KiN}}
  config:
    output.elasticsearch:
      index: "vpndetect-%{[agent.version]}"
      username: "username"
      password: "password"
    filebeat.inputs:
    - type: filestream
      id: vpn
      paths:
      - /usr/share/vpnip/ip.txt
    setup:
      kibana:
        username: "username"
        password: "password"
      template.name: "vpndetect"
      template.pattern: "vpndetect-%{[agent.version]}"
      ilm:
        check_exists: true
        enabled: true
        overwrite: false
        policy_file: /usr/share/filebeat/rsiem-filebeat-lifecycle-policy.json
        policy_name: filebeat-lifecycle-policy
        rollover_alias: vpndetect
    http.enabled: true
    http.port: {{.Values.Ports.vpn}}
    http.host: 0.0.0.0
    monitoring.enabled: false
  deployment:
    replicas: 1
    podTemplate:
      metadata:
        labels:
          stack-monitoring.elastic.co/type: beat
        annotations:
          co.elastic.logs/enabled: "true"
      spec:
        nodeName: worker-01
        securityContext:
          runAsUser: 0
        containers:
        - name: filebeat
          ports:
          - containerPort: {{.Values.Ports.vpn}}
            name: monitoring
            protocol: TCP
          volumeMounts:
          - mountPath: usr/share/filebeat/rsiem-filebeat-lifecycle-policy.json
            name: mod
            subPath: rsiem-filebeat-lifecycle-policy.json
          - mountPath: /usr/share/vpnip
            name: host-mount
        volumes:
        - name: mod
          configMap:
            name: filebeat-{{.Values.Kubernetes.Namespace}}
        - name: host-mount
          hostPath:
            path: /home/rs/vpn                                       

now here is the role config (a simple "vpndetect*" and an "all" privillage would have worked i'll change this later):

and this is the user: