GKE Elastic Agent, Ingest pipeline permissions question

Elastic Stack Elasticsearch
1

Hi All,

I have elastic cloud, with the elastic-agent deployed in a kubernetes cluster, this has a GCP integration to get messages from pubsub.

Once the agent gets the messages from pubsub i have an ingest pipeline that has a script processor in it to use a specific index e.g. my-index-yyyy.mm.dd

if the index doesnt exist it tries to create it but i get the error:

Cannot index event (status=403): dropping event!

I enabled debug logging and get this error:

action [indices:admin/auto_create] is unauthorized for API key id [123123123] of user [elastic/fleet-server] on indices [my-index-2024.11.11], this action is granted by the index privileges [auto_configure,create_index,manage,all]

I cant for the life of me figure out where i can assign these privileges so that it is able to create the index, does anyone know?

2

If I'm not wrong you cannot use an Elastic Agent managed by Fleet to write into a custom index like that.

Check this answer to a similar question.

No, it is not possible. The fleet-server service account is only able to write to indices that are managed by fleet, and there is no way to change that.

The alternative would be to write into something that the service account has permission, for example a data stream starting with logs-*.