Permissions issues after upgrading heartbeat to 8.0

Elastic Stack Beats
1

Good Day,

I recently upgrade my test installation of the Elastic Stack to 8.0. The Elasticsearch and kibana installations went fine. On the heartbeat side I'm getting the following errors:

Cannot index event publisher.Event {removed data} \"reason\":\"action [indices:admin/mapping/auto_put] is unauthorized for user [heartbeat_writer] with roles [heartbeat_writer] on indices [.ds-heartbeat-8.0.0-2022.02.16-000001], this action is granted by the index privileges [auto_configure,manage,write,all]\"}, dropping event!","service.name":"heartbeat","ecs.version":"1.6.0"

I updated through RPM package since my system is offline (no internet access). Just in case I've re-run the heartbeat setup with my yml file with a user (elastic) which has all privileges. I've set my heartbeat-writer role with the permissions explained in the heartbeat docs:
https://www.elastic.co/guide/en/beats/heartbeat/current/privileges-to-publish-events.html

My yml is very simple. I kept defaults but for the Elasticsearch output for the server ip, user/password. All my monitors are configured in the monitors.d folder and should not have any impact on this topic.

There's no index "heartbeat-8.0.0" that has created but a ".ds-heartbeat-8.0.0" appeared with just a few documents. 53 for now (for the last 10 hours) and I used to have 1656 docs per day before the upgrade. The current docs are all timeouts probably when I was doing firewall reloads.

Did I miss that now the index naming is .ds-beat-version etc?

Thanks,

2

Apologies for the poor experience @Marc-Antoine_J , and another apology for the delay in responding here.

We can do better here, this is a result of this breaking change in 8.0, where we now store events in data streams by default.

Data streams really are superior, but you'll need to modify your permissions to match the the error message shown. You can do that by adding the manage_index_templates cluster permission to the hearbeat_writer role.

I've opened beats#30567 this PR which fixes the docs as well.

3

I should mention that you don't really need to change the index level perms to account for the .ds-* prefix, that's an internal detail that's hidden behind the datastream abstraction in most cases, but leaks through in some places.

4

Hi @Andrew_Cholakian1 , thanks for trying to help. It seems that I had tried to add "manage_index_templates" to my role. This morning I've tried to restart my heartbeat service just to make sure the permissions are applied correctly but I still experience the exact same issues.

So if I understand correctly my role still has to apply permissions on heartbeat-* indexes and does not need to apply any on .ds-* since this is all in the datastream abstraction.

Any clues about why I still have the issue?

Thanks!

5

Here's a JSON formatted journalctl log with maybe more information. Hope this helps:

Feb 24 09: 44: 52 --Server Name -- heartbeat[465243]: {
    "log.level": "warn",
    "@timestamp": "2022-02-24T09:44:52.256-0500",
    "log.logger": "elasticsearch",
    "log.origin": {
        "file.name": "elasticsearch/client.go",
        "file.line": 414
    },
    "message": "Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2022, time.February, 24, 9, 44, 51, 132645113, time.Local), Meta:null, Fields:{--Removed Data--}}, Private:interface {}(nil), TimeSeries:false}, Flags:0x0, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=403): {\"type\":\"security_exception\",\"reason\":\"action [indices:admin/mapping/auto_put] is unauthorized for user [heartbeat_writer] with roles [heartbeat_writer] on indices [.ds-heartbeat-8.0.0-2022.02.16-000001], this action is granted by the index privileges [auto_configure,manage,write,all]\"}, dropping event!",
    "service.name": "heartbeat",
    "ecs.version": "1.6.0"
}
6

That's really surprising, I tried replicating the issue myself yesterday and found that change did fix things. I got the error without that perm, and didn't when I added it.

Could you try adding those perms to .ds-heartbeat-* and let us know if that helps. It won't hurt anything, and it'd be interesting to see if that fixes it.

7

@Andrew_Cholakian1 thanks for taking the time to troubleshoot our issue. It did not change anything to add .ds-heartbeat-* to the role. I did not expect either since it seems a cluster permission issue more than an index permissions issue. Maybe I'm missing something on my role setup. from the log you can see the role used. I'm attaching a screenshot of the kibana view of the role setup.

heartbeat_writer_permissions
heartbeat_writer_permissions1222×736 43.1 KB

Thanks,

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

9

I think this fix will solve your issue: Put data stream so there is no need for additional permissions by kvch · Pull Request #31048 · elastic/beats · GitHub

1 Like