APM Server has still not connected to Elasticsearch"

[mdamera (Murali)]

Hi Guys,
We have installed APM server and config it to connect to my elastic,
when i click " APM Server status" button i get the following message: "APM Server has still not connected to Elasticsearch"

this is what I can see in apm server logs

2020-08-06T09:10:13.414-0700 INFO pipeline/output.go:93 Attempting to reconnect to backoff(elasticsearch(http://R*********:9200)) with 320 reconnect attempt(s)
2020-08-06T09:10:14.003-0700 INFO elasticsearch/client.go:739 Attempting to connect to Elasticsearch version 6.8.2
2020-08-06T09:10:14.005-0700 INFO template/load.go:81 Loading template for Elasticsearch version: 6.8.2
2020-08-06T09:10:48.916-0700 ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(http://R***********:9200)): Connection marked as failed because the onConnect callback failed: Error loading Elasticsearch template: could not load template. Elasticsearch returned: couldn't load template: couldn't load json. Error: 403 Forbidden: {"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:admin/template/put] is unauthorized for user [apm_system]"}],"type":"security_exception","reason":"action [indices:admin/template/put] is unauthorized for user [apm_system]"},"status":403}. Response body: {"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:admin/template/put] is unauthorized for user [apm_system]"}],"type":"security_exception","reason":"action [indices:admin/template/put] is unauthorized for user [apm_system]"},"status":403}. Template is: map[index_patterns:[apm-6.8.8-*]

Elasticsearch version:
"6.x"
APM Server version:
6.8

[felixbarny (Felix Barnsteiner)]

HI and welcome to the forum

Seems like you are using the apm_system user instead of the elastic user.
See also Insufficent permission for apm_system user

[mdamera (Murali)]

No luck, still I could see below error.

I dont have apm_system in apm-server.yml file.Not sure why its still showing

2020-08-11T10:03:35.273-0700 ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(http://R********:9200)): Connection marked as failed because the onConnect callback failed: Error loading Elasticsearch template: could not load template. Elasticsearch returned: couldn't load template: couldn't load json. Error: 403 Forbidden: {"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:admin/template/put] is unauthorized for user [apm_system]"}],"type":"security_exception","reason":"action [indices:admin/template/put] is unauthorized for user [apm_system]"},"status":403}. Response body: {"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:admin/template/put] is unauthorized for user [apm_system]"}],"type":"security_exception","reason":"action [indices:admin/template/put] is unauthorized for user [apm_system]"},"status":403}. Template is: map[order:%!s(int=1)

[simitt (Silvia Mitter)]

I tried to reproduce the behavior with APM Server version 6.8.8 but cannot reproduce.

When security is enabled in Elasticsearch you need to configure a user and password for APM Server connecting to Elasticsearch, either via apm-server.yml or via options you pass in, e.g. ./apm-server -E output.elasticsearch.username=xyz -E output.elasticsearch.password=foo. Please double check that you do not use the apm_system user when connecting to ES.

The only part where this user is set as default is for monitoring, but according to the error message the issue occurs when trying to load the index templates.