Insufficent permission for apm_system user

My APM server cannot connect to ElasticSearch using apm_system user and ,
I've tried disable and enabled setup.template.overwrite.

Kibana version:

Elasticsearch version:

APM Server version:

APM Agent language and version:

Browser version:

Original install method (e.g. download page, yum, deb, from source, etc.) and version: I downloaded zips and created Windows services

Fresh install or upgraded from other version? Fresh

Is there anything special in your setup?
I started with no X-Pack security enabled first, then hours later, I enabled security and created password for default system users/accounts
Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
APM server logs show that it cannot connect to ElasticSearch due to following logs.

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):
ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(http://localhost:9200)): Connection marked as failed because the onConnect callback failed: error loading Elasticsearch template: could not load template. Elasticsearch returned: couldn't load template: 403 Forbidden: {"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:admin/template/put] is unauthorized for user [apm_system]"}],"type":"security_exception","reason":"action [indices:admin/template/put] is unauthorized for user [apm_system]"},"status":403}. Response body: {"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:admin/template/put] is unauthorized for user [apm_system]"}],"type":"security_exception","reason":"action [indices:admin/template/put] is unauthorized for user [apm_system]"},"status":403}. Template is: {
"index_patterns": [

Hello and thanks for trying Elastic APM and welcome to the Elastic Discuss Forum.

Question ... how did you configure the Output section in the apm-server.yml did you use the built in elastic user or some other user?

#-------------------------- Elasticsearch output ------------------------------
  # Array of hosts to connect to.
  # Scheme and port can be left out and will be set to the default (http and 9200)
  # In case you specify and additional path, the scheme is required: http://localhost:9200/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:9200
  hosts: ["my-es-cluster:9200"]

  # Boolean flag to enable or disable the output module.
  #enabled: true

  # Set gzip compression level.
  #compression_level: 0

  # Optional protocol and basic auth credentials.
  protocol: "https"
  username: "elastic"
  password: "changeme"

Thank you,
I used apm_system built in user. So, according to your answer, I should use elastic user instead of apm_system, shouldn't I?

Yes apologies for not being clear use the elastic user not the apm_system user which is typically used for monitoring the APM Server not for ingesting APM events. See built in roles here

Thank you very much for your reply.

I’m sure that it is going to work well.

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.