I've set up an Elastic Stack 7.9.2 and enabled basic security.
Metricbeat can not use to Elasticsearch user beats_system because of a lack of permissions.
The documentation Built-in users | Elasticsearch Guide [8.11] | Elastic says:
beats_system
The user the Beats use when storing monitoring information in Elasticsearch.remote_monitoring_user
The user Metricbeat uses when collecting and storing monitoring information in Elasticsearch. It has the remote_monitoring_agent and remote_monitoring_collector built-in roles.
However I have the following errors:
2020-10-27T06:03:35.433Z ERROR [publisher_pipeline_output] pipeline/output.go:154 Failed to connect to backoff(elasticsearch(https://myelasticsearch:9200)): Connection marked as failed because the onConnect callback failed: failed to check for policy name 'metricbeat': (status=403) {"error":{"root_cause":[{"type":"security_exception","reason":"action [cluster:admin/ilm/get] is unauthorized for user [beats_system]"}],"type":"security_exception","reason":"action [cluster:admin/ilm/get] is unauthorized for user [beats_system]"},"status":403}: 403 Forbidden: {"error":{"root_cause":[{"type":"security_exception","reason":"action [cluster:admin/ilm/get] is unauthorized for user [beats_system]"}],"type":"security_exception","reason":"action [cluster:admin/ilm/get] is unauthorized for user [beats_system]"},"status":403}
2020-10-27T06:17:55.149Z WARN [elasticsearch] elasticsearch/client.go:407 Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xbfde0cc473f68322, ext:21273920703, loc:(*time.Location)(0x7f56740)}, Meta:{"index":".monitoring-es-7-mb"}, Fields:{"agent":{"ephemeral_id":"b1371185-8cb2-4405-8dc3-eceab3497699","hostname":"myelasticsearch.eu-west-3.compute.internal","id":"23c7bc6d-bbca-42b7-bed6-d7724ea57721","name":"myelasticsearch","type":"metricbeat","version":"7.9.2"},"cluster_uuid":"FePunFMCQo-kWLmWe5dwPw","ecs":{"version":"1.5.0"},"event":{"dataset":"elasticsearch.index","duration":670459007,"module":"elasticsearch"},"host":{"architecture":"x86_64","containerized":false,"hostname":"myelasticsearch.eu-west-3.compute.internal","id":"cab9605edaa5484da7c2f02b8fd10762","ip":["xxx","xxx"],"mac":["xxx"],"name":"myelasticsearch","os":{"codename":"Core","family":"redhat","kernel":"3.10.0-1127.el7.x86_64","name":"CentOS Linux","platform":"centos","version":"7 (Core)"}},"index_stats":{"created":1599818418911,"hidden":false,"index":".kibana_task_manager_1","primaries":{"docs":{"count":6},"fielddata":{"evictions":0,"memory_size_in_bytes":0},"indexing":{"index_time_in_millis":262,"index_total":78,"throttle_time_in_millis":0},"merges":{"total_size_in_bytes":613493},"query_cache":{"evictions":0,"hit_count":0,"memory_size_in_bytes":0,"miss_count":0},"refresh":{"external_total_time_in_millis":1734,"total_time_in_millis":1589},"request_cache":{"evictions":0,"hit_count":0,"memory_size_in_bytes":0,"miss_count":1},"search":{"query_time_in_millis":1476,"query_total":501},"segments":{"count":2,"doc_values_memory_in_bytes":2560,"fixed_bit_set_memory_in_bytes":96,"index_writer_memory_in_bytes":0,"memory_in_bytes":9040,"norms_memory_in_bytes":384,"points_memory_in_bytes":0,"stored_fields_memory_in_bytes":1040,"term_vectors_memory_in_bytes":0,"terms_memory_in_bytes":5056,"version_map_memory_in_bytes":0},"store":{"size_in_bytes":92759}},"shards":{"active_primaries":1,"active_replicas":0,"active_total":1,"initializing":0,"primaries":1,"relocating":0,"replicas":0,"total":1,"unassigned_primaries":0,"unassigned_replicas":0,"unassigned_total":0},"status":"green","total":{"docs":{"count":6},"fielddata":{"evictions":0,"memory_size_in_bytes":0},"indexing":{"index_time_in_millis":262,"index_total":78,"throttle_time_in_millis":0},"merges":{"total_size_in_bytes":613493},"query_cache":{"evictions":0,"hit_count":0,"memory_size_in_bytes":0,"miss_count":0},"refresh":{"external_total_time_in_millis":1734,"total_time_in_millis":1589},"request_cache":{"evictions":0,"hit_count":0,"memory_size_in_bytes":0,"miss_count":1},"search":{"query_time_in_millis":1476,"query_total":501},"segments":{"count":2,"doc_values_memory_in_bytes":2560,"fixed_bit_set_memory_in_bytes":96,"index_writer_memory_in_bytes":0,"memory_in_bytes":9040,"norms_memory_in_bytes":384,"points_memory_in_bytes":0,"stored_fields_memory_in_bytes":1040,"term_vectors_memory_in_bytes":0,"terms_memory_in_bytes":5056,"version_map_memory_in_bytes":0},"store":{"size_in_bytes":92759}},"uuid":"YlQ_LnciQsabzkpR44Spqg"},"interval_ms":10000,"metricset":{"name":"index","period":10000},"service":{"address":"myelasticsearch:9200","type":"elasticsearch"},"timestamp":"2020-10-27T06:17:54.542Z","type":"index_stats"}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=403): {"type":"security_exception","reason":"action [indices:admin/auto_create] is unauthorized for user [beats_system]"}
I've stumble upon this post describing similar issue on Elastic Stack 7.7
And the role documentation says the contrary of above user documentation
beats_system
Grants access necessary for the Beats system user to send system-level data (such as monitoring) to Elasticsearch.
This role should not be assigned to users as the granted permissions may change between releases.
This role does not provide access to the beats indices and is not suitable for writing beats output to Elasticsearch.remote_monitoring_agent
Grants the minimum privileges required to write data into the monitoring indices ( .monitoring-* ). This role also has the privileges necessary to create Metricbeat indices ( metricbeat-* ) and write data into them.
So I have added another user with both beats_system and remote_monitoring_agent roles. With this user, metric reporting is alright, but I still can not load dashboards into Kibana
$ metricbeat setup
Exiting: error loading config file: open /etc/metricbeat/metricbeat.yml: permission denied
[centos@ip-10-0-10-10 metricbeat]$ sudo metricbeat setup
Overwriting ILM policy is disabled. Set `setup.ilm.overwrite: true` for enabling.
Index setup finished.
Loading dashboards (Kibana must be running and reachable)
Exiting: 1 error: error loading index pattern: returned 500 to import file: <nil>. Response: {"statusCode":500,"error":"Internal Server Error","message":"An internal server error occurred."}
{"type":"log","@timestamp":"2020-10-27T07:28:19Z","tags":["error","http"],"pid":8486,"message":"{ Error: Unable to bulk_create index-pattern\n at SecureSavedObjectsClientWrapper.ensureAuthorized (/usr/share/kibana/x-pack/plugins/security/server/saved_objects/secure_saved_objects_client_wrapper.js:194:48)\n at process._tickCallback (internal/process/next_tick.js:68:7)\n isBoom: true,\n isServer: false,\n data: null,\n output:\n { statusCode: 403,\n payload:\n { statusCode: 403,\n error: 'Forbidden',\n message: 'Unable to bulk_create index-pattern' },\n headers: {} },\n reformat: [Function],\n [Symbol(SavedObjectsClientErrorCode)]: 'SavedObjectsClient/forbidden' }"}
{"type":"error","@timestamp":"2020-10-27T07:28:19Z","tags":[],"pid":8486,"level":"error","error":{"message":"Internal Server Error","name":"Error","stack":"Error: Internal Server Error\n at HapiResponseAdapter.toInternalError (/usr/share/kibana/src/core/server/http/router/response_adapter.js:69:19)\n at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:170:34)\n at process._tickCallback (internal/process/next_tick.js:68:7)"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":"?force=true","query":{"force":"true"},"pathname":"/api/kibana/dashboards/import","path":"/api/kibana/dashboards/import?force=true","href":"/api/kibana/dashboards/import?force=true"},"message":"Internal Server Error"}
How should I configure Beats when using a secured Elasticsearch? Which user should I use for output.elasticsearch, setup.kibana and module: elasticsearch sections?