# Beats\_system role

**URL:** <https://discuss.elastic.co/t/beats-system-role/253391>\
**Category:** Beats\
**Tags:** elastic-stack-security, metricbeat\
**Created:** [October 27, 2020, 6:36am UTC](https://discuss.elastic.co/t/beats-system-role/253391 "2020-10-27T06:36:28Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![gquintana](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gquintana/32/48441_2.png) [@gquintana](https://discuss.elastic.co/u/gquintana)\
**Post date:** [October 27, 2020, 6:36am UTC](https://discuss.elastic.co/t/beats-system-role/253391/1 "2020-10-27T06:36:29Z")

</div>

I've set up an Elastic Stack 7.9.2 and enabled basic security.

Metricbeat can not use to Elasticsearch user beats\_system because of a lack of permissions.  
The documentation [Built-in users | Elasticsearch Guide [8.11] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/built-in-users.html#built-in-users) says:

> beats\_system  
> The user the Beats use when storing monitoring information in Elasticsearch.
> 
> remote\_monitoring\_user  
> The user Metricbeat uses when collecting and storing monitoring information in Elasticsearch. It has the remote\_monitoring\_agent and remote\_monitoring\_collector built-in roles.

However I have the following errors:

```auto
    2020-10-27T06:03:35.433Z	ERROR	[publisher_pipeline_output]	pipeline/output.go:154	Failed to connect to backoff(elasticsearch(https://myelasticsearch:9200)): Connection marked as failed because the onConnect callback failed: failed to check for policy name 'metricbeat': (status=403) {"error":{"root_cause":[{"type":"security_exception","reason":"action [cluster:admin/ilm/get] is unauthorized for user [beats_system]"}],"type":"security_exception","reason":"action [cluster:admin/ilm/get] is unauthorized for user [beats_system]"},"status":403}: 403 Forbidden: {"error":{"root_cause":[{"type":"security_exception","reason":"action [cluster:admin/ilm/get] is unauthorized for user [beats_system]"}],"type":"security_exception","reason":"action [cluster:admin/ilm/get] is unauthorized for user [beats_system]"},"status":403}

    2020-10-27T06:17:55.149Z	WARN	[elasticsearch]	elasticsearch/client.go:407	Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfde0cc473f68322, ext:21273920703, loc:(*time.Location)(0x7f56740)}, Meta:{"index":".monitoring-es-7-mb"}, Fields:{"agent":{"ephemeral_id":"b1371185-8cb2-4405-8dc3-eceab3497699","hostname":"myelasticsearch.eu-west-3.compute.internal","id":"23c7bc6d-bbca-42b7-bed6-d7724ea57721","name":"myelasticsearch","type":"metricbeat","version":"7.9.2"},"cluster_uuid":"FePunFMCQo-kWLmWe5dwPw","ecs":{"version":"1.5.0"},"event":{"dataset":"elasticsearch.index","duration":670459007,"module":"elasticsearch"},"host":{"architecture":"x86_64","containerized":false,"hostname":"myelasticsearch.eu-west-3.compute.internal","id":"cab9605edaa5484da7c2f02b8fd10762","ip":["xxx","xxx"],"mac":["xxx"],"name":"myelasticsearch","os":{"codename":"Core","family":"redhat","kernel":"3.10.0-1127.el7.x86_64","name":"CentOS Linux","platform":"centos","version":"7 (Core)"}},"index_stats":{"created":1599818418911,"hidden":false,"index":".kibana_task_manager_1","primaries":{"docs":{"count":6},"fielddata":{"evictions":0,"memory_size_in_bytes":0},"indexing":{"index_time_in_millis":262,"index_total":78,"throttle_time_in_millis":0},"merges":{"total_size_in_bytes":613493},"query_cache":{"evictions":0,"hit_count":0,"memory_size_in_bytes":0,"miss_count":0},"refresh":{"external_total_time_in_millis":1734,"total_time_in_millis":1589},"request_cache":{"evictions":0,"hit_count":0,"memory_size_in_bytes":0,"miss_count":1},"search":{"query_time_in_millis":1476,"query_total":501},"segments":{"count":2,"doc_values_memory_in_bytes":2560,"fixed_bit_set_memory_in_bytes":96,"index_writer_memory_in_bytes":0,"memory_in_bytes":9040,"norms_memory_in_bytes":384,"points_memory_in_bytes":0,"stored_fields_memory_in_bytes":1040,"term_vectors_memory_in_bytes":0,"terms_memory_in_bytes":5056,"version_map_memory_in_bytes":0},"store":{"size_in_bytes":92759}},"shards":{"active_primaries":1,"active_replicas":0,"active_total":1,"initializing":0,"primaries":1,"relocating":0,"replicas":0,"total":1,"unassigned_primaries":0,"unassigned_replicas":0,"unassigned_total":0},"status":"green","total":{"docs":{"count":6},"fielddata":{"evictions":0,"memory_size_in_bytes":0},"indexing":{"index_time_in_millis":262,"index_total":78,"throttle_time_in_millis":0},"merges":{"total_size_in_bytes":613493},"query_cache":{"evictions":0,"hit_count":0,"memory_size_in_bytes":0,"miss_count":0},"refresh":{"external_total_time_in_millis":1734,"total_time_in_millis":1589},"request_cache":{"evictions":0,"hit_count":0,"memory_size_in_bytes":0,"miss_count":1},"search":{"query_time_in_millis":1476,"query_total":501},"segments":{"count":2,"doc_values_memory_in_bytes":2560,"fixed_bit_set_memory_in_bytes":96,"index_writer_memory_in_bytes":0,"memory_in_bytes":9040,"norms_memory_in_bytes":384,"points_memory_in_bytes":0,"stored_fields_memory_in_bytes":1040,"term_vectors_memory_in_bytes":0,"terms_memory_in_bytes":5056,"version_map_memory_in_bytes":0},"store":{"size_in_bytes":92759}},"uuid":"YlQ_LnciQsabzkpR44Spqg"},"interval_ms":10000,"metricset":{"name":"index","period":10000},"service":{"address":"myelasticsearch:9200","type":"elasticsearch"},"timestamp":"2020-10-27T06:17:54.542Z","type":"index_stats"}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=403): {"type":"security_exception","reason":"action [indices:admin/auto_create] is unauthorized for user [beats_system]"}

```

I've stumble upon this [post](https://discuss.elastic.co/t/setting-up-a-lab-as-prod-beats-roles-permissions-and-tutorial/236624) describing similar issue on Elastic Stack 7.7

And the role documentation says the contrary of above user documentation

> **[Built-in roles | Elasticsearch Guide \[7.9\] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/7.9/built-in-roles.html)**

> beats\_system  
> Grants access necessary for the Beats system user to send system-level data (such as monitoring) to Elasticsearch.  
> This role should not be assigned to users as the granted permissions may change between releases.  
> This role does not provide access to the beats indices and is not suitable for writing beats output to Elasticsearch.
> 
> remote\_monitoring\_agent  
> Grants the minimum privileges required to write data into the monitoring indices ( .monitoring-\* ). This role also has the privileges necessary to create Metricbeat indices ( metricbeat-\* ) and write data into them.

So I have added another user with both beats\_system and remote\_monitoring\_agent roles. With this user, metric reporting is alright, but I still can not load dashboards into Kibana

```auto
$ metricbeat setup
Exiting: error loading config file: open /etc/metricbeat/metricbeat.yml: permission denied
[centos@ip-10-0-10-10 metricbeat]$ sudo metricbeat setup
Overwriting ILM policy is disabled. Set `setup.ilm.overwrite: true` for enabling.

Index setup finished.
Loading dashboards (Kibana must be running and reachable)
Exiting: 1 error: error loading index pattern: returned 500 to import file: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred."}
{"type":"log","@timestamp":"2020-10-27T07:28:19Z","tags":["error","http"],"pid":8486,"message":"{ Error: Unable to bulk_create index-pattern\n at SecureSavedObjectsClientWrapper.ensureAuthorized (/usr/share/kibana/x-pack/plugins/security/server/saved_objects/secure_saved_objects_client_wrapper.js:194:48)\n at process._tickCallback (internal/process/next_tick.js:68:7)\n isBoom: true,\n isServer: false,\n data: null,\n output:\n { statusCode: 403,\n payload:\n { statusCode: 403,\n error: 'Forbidden',\n message: 'Unable to bulk_create index-pattern' },\n headers: {} },\n reformat: [Function],\n [Symbol(SavedObjectsClientErrorCode)]: 'SavedObjectsClient/forbidden' }"}
{"type":"error","@timestamp":"2020-10-27T07:28:19Z","tags":[],"pid":8486,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n at HapiResponseAdapter.toInternalError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:69:19)\n at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:170:34)\n at process._tickCallback (internal/process/next_tick.js:68:7)"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":"?force=true","query":{"force":"true"},"pathname":"/api/kibana/dashboards/import","path":"/api/kibana/dashboards/import?force=true","href":"/api/kibana/dashboards/import?force=true"},"message":"Internal Server Error"}

```

How should I configure Beats when using a secured Elasticsearch? Which user should I use for `output.elasticsearch`, `setup.kibana` and `module: elasticsearch` sections?

---

<div class="post-metadata">

**Author:** ![ChrsMark](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrsmark/32/55858_2.png) [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)\
**Post date:** [October 27, 2020, 10:00am UTC](https://discuss.elastic.co/t/beats-system-role/253391/2 "2020-10-27T10:00:32Z")

</div>

Hi!

I would try with `kibana_system` for setting up Kibana and maybe for `elasticsearch` module too.

---

<div class="post-metadata">

**Author:** ![gquintana](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gquintana/32/48441_2.png) [@gquintana](https://discuss.elastic.co/u/gquintana)\
**Post date:** [October 27, 2020, 10:54am UTC](https://discuss.elastic.co/t/beats-system-role/253391/3 "2020-10-27T10:54:50Z")

</div>

Neither `kibana_system` nor `kibana_admin` are enough. My current workaround is to use the `elastic` user, but it's ugly.

---

<div class="post-metadata">

**Author:** ![ChrsMark](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chrsmark/32/55858_2.png) [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)\
**Post date:** [October 27, 2020, 11:51am UTC](https://discuss.elastic.co/t/beats-system-role/253391/4 "2020-10-27T11:51:21Z")

</div>

I see. Can you redirect your issue/concern to the Elasticsearch forum? Maybe there is the need to provide more options here for such cases.

---

<div class="post-metadata">

**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\
**Post date:** [October 28, 2020, 3:25am UTC](https://discuss.elastic.co/t/beats-system-role/253391/5 "2020-10-28T03:25:49Z")

</div>

The builtin users and roles aren't really intended for this purpose. The Metricbeat security guide has the steps you need.

> **[Secure Metricbeat | Metricbeat Reference \[7.9\] | Elastic](https://www.elastic.co/guide/en/beats/metricbeat/current/securing-metricbeat.html)**

---

<div class="post-metadata">

**Author:** ![gquintana](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gquintana/32/48441_2.png) [@gquintana](https://discuss.elastic.co/u/gquintana)\
**Post date:** [October 29, 2020, 1:19pm UTC](https://discuss.elastic.co/t/beats-system-role/253391/6 "2020-10-29T13:19:38Z")

</div>

Thanks for the pointer to the documentation.  
However, even in this page [Grant users access to secured resources | Metricbeat Reference [8.11] | Elastic](https://www.elastic.co/guide/en/beats/metricbeat/current/feature-roles.html), the documentation is confusing:

> Elasticsearch security features provides [built-in roles](https://www.elastic.co/guide/en/elasticsearch/reference/7.9/built-in-roles.html) that grant a subset of the privileges needed by Metricbeat users. When possible, use the built-in roles to minimize the affect of future changes on your security strategy.

If builtin users and roles are nor intended for this purpose, I wonder which one they are.

---

<div class="post-metadata">

**Author:** ![joaociocca](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/joaociocca/32/15827_2.png) [@joaociocca](https://discuss.elastic.co/u/joaociocca)\
**Post date:** [November 3, 2020, 5:07am UTC](https://discuss.elastic.co/t/beats-system-role/253391/7 "2020-11-03T05:07:18Z")

</div>

I'm still wondering till this day. Since you mentioned my post, I think you noticed how it got no replies at all. It's as I often say: Elastic does amazing tools, but their documentation expects people to already master the tools before they can understand the documentation.

Bottom line is: ignore what the documentation says, feel your way around and fix things as you go - specially when it comes to permissions.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 1, 2020, 7:07am UTC](https://discuss.elastic.co/t/beats-system-role/253391/8 "2020-12-01T07:07:19Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.