Insufficent permission for apm_system user

[chikien276]

My APM server cannot connect to ElasticSearch using apm_system user and ,
I've tried disable and enabled setup.template.overwrite.

Kibana version:
7.2.0

Elasticsearch version:
7.2.0

APM Server version:
7.2.0

APM Agent language and version:

Browser version:

Original install method (e.g. download page, yum, deb, from source, etc.) and version: I downloaded zips and created Windows services

Fresh install or upgraded from other version? Fresh

Is there anything special in your setup?
I started with no X-Pack security enabled first, then hours later, I enabled security and created password for default system users/accounts
Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
APM server logs show that it cannot connect to ElasticSearch due to following logs.

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):
ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(http://localhost:9200)): Connection marked as failed because the onConnect callback failed: error loading Elasticsearch template: could not load template. Elasticsearch returned: couldn't load template: 403 Forbidden: {"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:admin/template/put] is unauthorized for user [apm_system]"}],"type":"security_exception","reason":"action [indices:admin/template/put] is unauthorized for user [apm_system]"},"status":403}. Response body: {"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:admin/template/put] is unauthorized for user [apm_system]"}],"type":"security_exception","reason":"action [indices:admin/template/put] is unauthorized for user [apm_system]"},"status":403}. Template is: {
"index_patterns": [
"apm-7.2.0*"
],
...
}

[stephenb (Stephen Brown)]

Hello and thanks for trying Elastic APM and welcome to the Elastic Discuss Forum.

Question ... how did you configure the Output section in the apm-server.yml did you use the built in elastic user or some other user?

#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  # Scheme and port can be left out and will be set to the default (http and 9200)
  # In case you specify and additional path, the scheme is required: http://localhost:9200/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:9200
  hosts: ["my-es-cluster:9200"]

  # Boolean flag to enable or disable the output module.
  #enabled: true

  # Set gzip compression level.
  #compression_level: 0

  # Optional protocol and basic auth credentials.
  protocol: "https"
  username: "elastic"
  password: "changeme"

[chikien276]

Thank you,
I used apm_system built in user. So, according to your answer, I should use elastic user instead of apm_system, shouldn't I?

[stephenb (Stephen Brown)]

Yes apologies for not being clear use the elastic user not the apm_system user which is typically used for monitoring the APM Server not for ingesting APM events. See built in roles here

[chikien276]

Thank you very much for your reply.

I’m sure that it is going to work well.