Heartbeat setup failing because of incorrect user

Elastic Stack Beats
1

I'm trying to setup heartbeat on my new elastic stack server but it is failing because the user doesn't have permission to create indices.
It seems to be using the writer user instead of the setup user that has the correct permissions.
I used these same users to setup metricbeat and filebeat perfectly fine.

heartbeat.config.monitors:
  path: ${path.config}/monitors.d/*.yml
  reload.enabled: false
  reload.period: 5s
heartbeat.monitors:
setup.template.settings:
  index.number_of_shards: 1
  index.codec: best_compression
setup.kibana:
  host: "https://pacc-elk.pac.internal:5601"
  ssl.enabled: True
  username: "beats_setup"
  password: "pass"
output.elasticsearch:
  hosts: ["pacc-elk.pac.internal:9200"]
  protocol: "https"
  username: "beats_writer"
  password: "pass"
processors:
  - add_observer_metadata:

And Here is the error I'm getting

2021-08-10T14:21:21.619+0930	INFO	instance/beat.go:665	Home path: [/usr/share/heartbeat] Config path: [/etc/heartbeat] Data path: [/var/lib/heartbeat] Logs path: [/var/log/heartbeat]
2021-08-10T14:21:21.620+0930	INFO	instance/beat.go:673	Beat ID: d9c4ed47-62fd-42a3-917b-6624b46952f3
2021-08-10T14:21:21.621+0930	INFO	[beat]	instance/beat.go:1014	Beat info	{"system_info": {"beat": {"path": {"config": "/etc/heartbeat", "data": "/var/lib/heartbeat", "home": "/usr/share/heartbeat", "logs": "/var/log/heartbeat"}, "type": "heartbeat", "uuid": "d9c4ed47-62fd-42a3-917b-6624b46952f3"}}}
2021-08-10T14:21:21.621+0930	INFO	[beat]	instance/beat.go:1023	Build info	{"system_info": {"build": {"commit": "e127fc31fc6c00fdf8649808f9421d8f8c28b5db", "libbeat": "7.14.0", "time": "2021-07-29T21:02:53.000Z", "version": "7.14.0"}}}
2021-08-10T14:21:21.621+0930	INFO	[beat]	instance/beat.go:1026	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":2,"version":"go1.16.6"}}}
2021-08-10T14:21:21.622+0930	INFO	[beat]	instance/beat.go:1030	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2021-08-06T17:03:51+09:30","containerized":false,"name":"pacc-elk.pac.internal","ip":["127.0.0.1/8","::1/128","192.168.200.70/24","fe80::250:56ff:fead:ee8b/64"],"kernel_version":"4.18.0-305.10.2.el8_4.x86_64","mac":["00:50:56:ad:ee:8b"],"os":{"type":"linux","family":"","platform":"rocky","name":"Rocky Linux","version":"8.4 (Green Obsidian)","major":8,"minor":4,"patch":0},"timezone":"ACST","timezone_offset_sec":34200,"id":"13b77172f2354c77a0ab9261c6346a5a"}}}
2021-08-10T14:21:21.622+0930	INFO	[beat]	instance/beat.go:1059	Process info	{"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read","38","39"],"ambient":null}, "cwd": "/root", "exe": "/usr/share/heartbeat/bin/heartbeat", "name": "heartbeat", "pid": 28009, "ppid": 27497, "seccomp": {"mode":"disabled","no_new_privs":false}, "start_time": "2021-08-10T14:21:21.080+0930"}}}
2021-08-10T14:21:21.622+0930	INFO	instance/beat.go:309	Setup Beat: heartbeat; Version: 7.14.0
2021-08-10T14:21:21.622+0930	INFO	[index-management]	idxmgmt/std.go:184	Set output.elasticsearch.index to 'heartbeat-7.14.0' as ILM is enabled.
2021-08-10T14:21:21.623+0930	INFO	[esclientleg]	eslegclient/connection.go:100	elasticsearch url: https://pacc-elk.pac.internal:9200
2021-08-10T14:21:21.624+0930	INFO	[publisher]	pipeline/module.go:113	Beat name: pacc-elk.pac.internal
2021-08-10T14:21:21.624+0930	INFO	[esclientleg]	eslegclient/connection.go:100	elasticsearch url: https://pacc-elk.pac.internal:9200
2021-08-10T14:21:21.697+0930	INFO	[esclientleg]	eslegclient/connection.go:273	Attempting to connect to Elasticsearch version 7.14.0
2021-08-10T14:21:21.732+0930	INFO	[index-management]	idxmgmt/std.go:261	Auto ILM enable success.
2021-08-10T14:21:21.734+0930	INFO	[index-management.ilm]	ilm/std.go:160	ILM policy heartbeat exists already.
2021-08-10T14:21:21.734+0930	INFO	[index-management]	idxmgmt/std.go:401	Set setup.template.name to '{heartbeat-7.14.0 {now/d}-000001}' as ILM is enabled.
2021-08-10T14:21:21.734+0930	INFO	[index-management]	idxmgmt/std.go:406	Set setup.template.pattern to 'heartbeat-7.14.0-*' as ILM is enabled.
2021-08-10T14:21:21.734+0930	INFO	[index-management]	idxmgmt/std.go:440	Set settings.index.lifecycle.rollover_alias in template to {heartbeat-7.14.0 {now/d}-000001} as ILM is enabled.
2021-08-10T14:21:21.734+0930	INFO	[index-management]	idxmgmt/std.go:444	Set settings.index.lifecycle.name in template to {heartbeat {"policy":{"phases":{"hot":{"actions":{"rollover":{"max_age":"30d","max_size":"50gb"}}}}}}} as ILM is enabled.
2021-08-10T14:21:21.735+0930	INFO	template/load.go:229	Existing template will be overwritten, as overwrite is enabled.
2021-08-10T14:21:21.834+0930	INFO	template/load.go:132	Try loading template heartbeat-7.14.0 to Elasticsearch
2021-08-10T14:21:21.968+0930	INFO	template/load.go:124	Template with name "heartbeat-7.14.0" loaded.
2021-08-10T14:21:21.968+0930	INFO	[index-management]	idxmgmt/std.go:297	Loaded index template.
2021-08-10T14:21:21.972+0930	ERROR	[index-management.ilm]	ilm/std.go:128	Index Alias heartbeat-7.14.0 setup failed: failed to create alias: {"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:admin/aliases] is unauthorized for user [beats_writer] with roles [beats_writer,remote_monitoring_agent,remote_monitoring_collector] on indices [heartbeat-7.14.0-2021.08.10-000001,heartbeat-7.14.0], this action is granted by the index privileges [manage,all]"}],"type":"security_exception","reason":"action [indices:admin/aliases] is unauthorized for user [beats_writer] with roles [beats_writer,remote_monitoring_agent,remote_monitoring_collector] on indices [heartbeat-7.14.0-2021.08.10-000001,heartbeat-7.14.0], this action is granted by the index privileges [manage,all]"},"status":403}: 403 Forbidden: {"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:admin/aliases] is unauthorized for user [beats_writer] with roles [beats_writer,remote_monitoring_agent,remote_monitoring_collector] on indices [heartbeat-7.14.0-2021.08.10-000001,heartbeat-7.14.0], this action is granted by the index privileges [manage,all]"}],"type":"security_exception","reason":"action [indices:admin/aliases] is unauthorized for user [beats_writer] with roles [beats_writer,remote_monitoring_agent,remote_monitoring_collector] on indices [heartbeat-7.14.0-2021.08.10-000001,heartbeat-7.14.0], this action is granted by the index privileges [manage,all]"},"status":403}.
2021-08-10T14:21:21.972+0930	ERROR	instance/beat.go:989	Exiting: failed to create alias: {"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:admin/aliases] is unauthorized for user [beats_writer] with roles [beats_writer,remote_monitoring_agent,remote_monitoring_collector] on indices [heartbeat-7.14.0-2021.08.10-000001,heartbeat-7.14.0], this action is granted by the index privileges [manage,all]"}],"type":"security_exception","reason":"action [indices:admin/aliases] is unauthorized for user [beats_writer] with roles [beats_writer,remote_monitoring_agent,remote_monitoring_collector] on indices [heartbeat-7.14.0-2021.08.10-000001,heartbeat-7.14.0], this action is granted by the index privileges [manage,all]"},"status":403}: 403 Forbidden: {"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:admin/aliases] is unauthorized for user [beats_writer] with roles [beats_writer,remote_monitoring_agent,remote_monitoring_collector] on indices [heartbeat-7.14.0-2021.08.10-000001,heartbeat-7.14.0], this action is granted by the index privileges [manage,all]"}],"type":"security_exception","reason":"action [indices:admin/aliases] is unauthorized for user [beats_writer] with roles [beats_writer,remote_monitoring_agent,remote_monitoring_collector] on indices [heartbeat-7.14.0-2021.08.10-000001,heartbeat-7.14.0], this action is granted by the index privileges [manage,all]"},"status":403}
2

TL;DR Setup privileges need to be on the ES output too.

I can see why this is a bit confusing! Typically, when using a setup user you will have the setup process run on a different box than the writer process. The use case for having a setup user is one where you don't want a given beat to be able to have the enhanced privileges that are only needed during set up in the course of its day-to-day operations. For instance, you might want to have file be on every server you own but not give each of them index management privileges. You would have one machine that runs the setup process with set up privileges on both the elasticsearch output and kibana, and would only give writer privileges to the individual servers.

What this typically means is that you would have one box with set up privileges set for both elasticsearch and kibana and run the setup command when you do an upgrade, before you upgrade the nodes that actually run day-to-day. Those nodes would have writer permissions.

For heartbeat this is may be less important. Heartbeat typically is not on edge nodes that are not controlled by the system administrators. Leaking credentials on heartbeat nodes to users is probably not very important in most cases. You can still use the pattern of having a set up process run on a different box for releases but it's less important.

3

Ah Thank you,
That makes sense.

I was following the guide here
Set up basic security for the Elastic Stack plus secured HTTPS traffic | Elasticsearch Guide [7.14] | Elastic
Which said to set it up that way for each beat.

4

I have just updated the beat to use the setup user for both but I am still getting the error.

"type":"security_exception","reason":"action [indices:admin/aliases/get] is unauthorized for user [beats_setup] with roles [beats_setup,kibana_admin,ingest_admin,beats_writer,remote_monitoring_agent,remote_monitoring_collector] on indices [heartbeat-7.14.0], this action is granted by the index privileges [view_index_metadata,manage,all]"},"status":403}

But you can see here for the beats_writer role has the index permission shown
image
And the beats_setup role has these
image

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.