Heartbeat ask for unnecessary cluster privileges at startup

[iorfix]

Hi all,
I'm using Heartbeat 7.13.1.
I first setup templates and ILM policy with elastic superuser, and then I start heartbeat using a ristrected "apikey" user. However, I have this error at startup:

 {"log.level":"error","@timestamp":"2021-06-14T11:06:35.785+0200","log.logger":"index-management.ilm","log.origin":{"file.name":"ilm/std.go","file.line":166},"message":"ILM policy heartbeat creation failed: 403 Forbidden: {\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/ilm/put] is unauthorized for API key id [0e0c9nkBdKUEp4Eqeb4y] of user [apikeywriteruser], this action is granted by the cluster privileges [manage_ilm,manage,all]\"}],\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/ilm/put] is unauthorized for API key id [0e0c9nkBdKUEp4Eqeb4y] of user [apikeywriteruser], this action is granted by the cluster privileges [manage_ilm,manage,all]\"},\"status\":403}","ecs.version":"1.6.0"}

Note that I've the following configuration on heartbeat.yml:

setup.template.enabled: false
setup.ilm.check_exists: false
setup.ilm.overwrite: false
setup.ilm.enabled: true

and the policy is already loaded, since the setup command completed successfully

heartbeat setup -E setup.ilm.overwrite=true -E setup.template.enabled=true -E output.elasticsearch.username=elastic -E output.elasticsearch.password=***

I would avoid to give cluster permission to beat users, is there a better solution?
Note that I use the same pattern (and even the same key) with metricbeat modules (7.10.2) and they worked correctly.
Thank you

[Andrew_Cholakian1 (Andrew Cholakian)]

Have you tried this with metricbeat 7.13.1? Under the hood they use the same code for settings / ES connection

[iorfix]

Hi @Andrew_Cholakian1 ,
I confirm I have the same problem with metricbeat 7.13.1, while I haven't with (at least) 7.10.2, but probably also newer version

2021-06-14T13:58:06.788Z        INFO    [index-management]      idxmgmt/std.go:261      Auto ILM enable success.
2021-06-14T13:58:06.790Z        ERROR   [index-management.ilm]  ilm/std.go:166  ILM policy metricbeat creation failed: 403 Forbidden: {"error":{"root_cause":[{"type":"security_exception","reason":"action [cluster:admin/ilm/put] is unauthorized for user [apikeywriteruser] with roles [role_beats_cloud,role_apm_api_key,role_restapiplus_api_key], this action is granted by the cluster privileges [manage_ilm,manage,all]"}],"type":"security_exception","reason":"action [cluster:admin/ilm/put] is unauthorized for user [apikeywriteruser] with roles [role_beats_cloud,role_apm_api_key,role_restapiplus_api_key], this action is granted by the cluster privileges [manage_ilm,manage,all]"},"status":403}

[Andrew_Cholakian1 (Andrew Cholakian)]

Thanks a ton for helping us out there, I'll ping the relevant people, since I'm not super up to date on changes there.

[steffens (Steffen Siering)]

This is definitely a bug. With these settings the Beat should not need to talk with Elasticsearch. The error message suggests that the Beat is trying to overwrite the ILM policy. Can you file it on github please?

[iorfix]

Issue opened at:

Beats require unnecessary privileges at startup on 7.13