How to change permissions for fleet-managed Elastic Agents? [security_exception]

Elastic Stack Elastic Agent
1

Hello,

I am reaching out to you with an issue that has been concerning me for quite some time now.

Overview

I have enrolled a fleet-managed Elastic Agent on a machine and created an ingest pipeline in the Elastic Cluster. As part of this ingest pipeline, I am using a `Set Processor` to reset the `_index` field in order to distribute the ingested documents across different indices. However, I am now encountering an error in the Fleet Agent logs like this: 
"{"type":"security_exception","reason":"action [indices:data/write/bulk[s]] is unauthorized for API key id [<api_key_here>] of user [elastic/fleet-server] on indices [my-new-index-metrics], this action is granted by the index privileges [create_doc,create,delete,index,write,all]"}, dropping event!"

Details

The implementation of the Docker cluster (V8.14.3) was done very similarly to this guide here.
Here is the part of the code for setting up the Fleet server:

Fleet-Server part of docker-compose 
  fleet-server:
    depends_on:
      kibana:
        condition: service_healthy
      es01:
        condition: service_healthy
    image: docker.elastic.co/beats/elastic-agent:${STACK_VERSION}
    volumes:
      - certs:/certs
      - fleetserverdata:/usr/share/elastic-agent
      - "/var/lib/docker/containers:/var/lib/docker/containers:ro"
      - "/var/run/docker.sock:/var/run/docker.sock:ro"
      - "/sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro"
      - "/proc:/hostfs/proc:ro"
      - "/:/hostfs:ro"
    ports:
      - ${FLEET_PORT}:8220
      - ${APMSERVER_PORT}:8200
    user: root
    environment:
      - SSL_CERTIFICATE_AUTHORITIES=/certs/ca/ca.crt
      - CERTIFICATE_AUTHORITIES=/certs/ca/ca.crt
      - FLEET_CA=/certs/ca/ca.crt
      - FLEET_ENROLL=1
      - FLEET_INSECURE=true
      - FLEET_SERVER_ELASTICSEARCH_CA=/certs/ca/ca.crt
      - FLEET_SERVER_ELASTICSEARCH_HOST=https://es01:9200
      - FLEET_SERVER_ELASTICSEARCH_INSECURE=true
      - FLEET_SERVER_ENABLE=true
      - FLEET_SERVER_CERT=/certs/fleet-server/fleet-server.crt
      - FLEET_SERVER_CERT_KEY=/certs/fleet-server/fleet-server.key
      - FLEET_SERVER_INSECURE_HTTP=true
      - FLEET_SERVER_POLICY_ID=fleet-server-policy
      - FLEET_URL=https://fleet-server:8220
      - KIBANA_FLEET_CA=/certs/ca/ca.crt
      - KIBANA_FLEET_SETUP=1
      - KIBANA_FLEET_USERNAME=elastic
      - KIBANA_FLEET_PASSWORD=${ELASTIC_PASSWORD}
      - KIBANA_HOST=https://kibana:5601

Everything is running very well, and I was able to successfully enroll the Elastic Agent on the machine. The status is healthy, and machine metrics are being ingested.

Attempts so far

  • Created an elastic/fleet-server user manually to change the permissions. I was able to edit the permissions but i was not allowed to save the changes. I now know that this was the wrong approach because this user is built-in and should not be modified.

  • Tried editing the elastic-agent.yml to set a newly generated API key or a newly created user with the appropriate permissions for the Elasticsearch output. However, the .yml file is ignored by the fleet-managed Elastic Agent. Is it possible that these settings have no effect on a fleet-managed Elastic Agent and only apply to a standalone Elastic Agent?

outputs:
  default:
    type: elasticsearch
    hosts: [https://fleet-server:9200]
    #api_key: "example-key"
    username: "elastic"
    password: "changeme"
  • I found a post about a similar issue here. The solution there was to rename the system.yml file to system.yml.disabled. This doesn't work for me.
  • Instead of the Set processor I tried using a reroute processor. However, this results in the exact same error message. I found the following in the official documentation for the reroute processor:
    grafik
    grafik750×162 5.97 KB

    Unfortunately, there were no additional solutions provided for this issue.

The big question now is how do i set the clients (fleet-managed Elastic Agent) permission or change the user or API key?

2

Anyone who can help with this?

3

I haven't found a solution to the mentioned problem yet.
Have I perhaps forgotten to mention important information to describe the initial situation?

I'm looking forward to an answer from someone.

4

What is the index name that you are using in your set processor?

I'm not sure, but I think that if the name is not one of the default names, like logs-* or metrics-* it will not work.

Check this similar question.

The documentation is a little confusing here, as it gives the impression that you could change the permissions for the elastic/fleet-server, but I don't think this is possible, it looks like this was a poor choice for an example.

@TimV Sorry to tag you, but since you answered a similar question, can you provide a little feedback here?

5

Tank you Leandro for your answer.

Your are right, the set processor should rename the _index field to something like <applicationname>-v<versionnumber>-telegram.
I haven't found any documentation that says the field or index has to use logs-*.
On the contrary, I remember reading somewhere that dividing data into different indices should work exactly like that by modifying the _index field.

I know that the elastic/fleet-server user cannot or should not be modified, so I have given up on any efforts in that direction.

After reading the linked question, I'm now unsure whether the set processor is the right tool for my task. We have many indices, which also have different retention times (ILM). Additionally, the index mappings would become far too large if we were to ingest everything into just one index. That's why we want to split our data across multiple indices.

What I also tested is the following: I can run the Elastic Agent in standalone mode and assign a user there. This way, the ingest pipeline with the included set processor works, allowing me to modify the _index field, and the data is successfully split.

6

Because it is not exactly a requirement, at least not in the majority of the cases, the issue here is specific when using Fleet managed Elastic Agent.

Since Fleet managed Elastic Agent uses the service token elastic/fleet-server, it can only write in the indices where this service token has permissions, so this would limit the set processor in this case to only be able to write to logs-* or metrics-*.

You still can have different retention times, but it is a little more complex to configure it, you need to follow the steps on this documentation.

When using Elastic Agent, specially Fleet managed, you would split your indices using the namespace.

And the customizations like mappings, processors and policies are done per dataset, so for one integration you may need to customize a lot of files.

This is being improved, but it still requires a lot of work.

Yeah, this would work, standalone mode is considered an advanced case when you need more control over the agent, it may be your case.

When using Fleet managed agents you do not have much control over anything and customization are complicated.

7

Thank you for your explanations. Regarding the use of dataset and namespace I will need to consider whether they can be useful for our purposes.

It now seems that the configuration and implementation for our purposes differ significantly between a standalone agent and a fleet-managed agent. I find this somewhat unfortunate, as we operate many agents that previously ran simple Filebeat and Metricbeat instances. We probably won’t be able to take advantage of Fleet’s benefits because the Set processor and Reroute processor simply aren’t compatible with Fleet.