APM agent fails to connect to APM server

If you are asking about a problem you are experiencing, please use the following template, as it will help us help you. If you have a different problem, please delete all of this text :slight_smile:

TIP 1: select at least one tag that further categorizes your topic. For example server for APM Server related questions, java for questions regarding the Elastic APM Java agent, or ui for questions about the APM App within Kibana.

TIP 2: Check out the troubleshooting guide first. Not only will it help you to resolve common problems faster but it also explains in more detail which information we need before we can properly help you.

Kibana version: 7.17

Elasticsearch version: 7.17

APM Server version: 7.17

APM Agent language and version: c#, .net 5, 7.17

Browser version: Edge

Original install method (e.g. download page, yum, deb, from source, etc.) and version: NA

Fresh install or upgraded from other version?

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc.
ElasticCloud (trial version), c# asp.net core, .net5 core app, Elastic.Apm.AspNetCore is added.

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
I have defined the APM settings in appsettings.json as shown below.

"ElasticApm": {
"ServerUrl": "https://575496a0a6cc4dd088c1158558776003.apm.southeastasia.azure.elastic-cloud.com:443",
"SecretToken": "KScUc3zsCPbt5er3H7",
"Environment": "production",
"ServiceName": "MyApp",
"TransactionSampleRate": 1.0,
"LogLevel": "Trace"
}

i got the url from elastic.co portal (active deployment), APM page.
Copied the token as well from the 'Edit Deployments -> APM & Fleet' section.

Created a Asp.net webapi application, instrumented the code for 1 transaction & 3 spans.
When i start the application, i am seeing this error in output.

Elastic.Apm: Error: {CentralConfigurationFetcher} Exception was thrown while fetching configuration from APM Server and parsing it. ETag: `'. URL: `https://575496a0a6cc4dd088c1158558776003.apm.southeastasia.azure.elastic-cloud.com/config/v1/agents?service.name=MyApp&service.environment=production'. Apm Server base URL: `https://575496a0a6cc4dd088c1158558776003.apm.southeastasia.azure.elastic-cloud.com/'. WaitInterval: 5m. dbgIterationsCount: 1. +-> Request: Method: GET, RequestUri: 'https://575496a0a6cc4dd088c1158558776003.apm.southeastasia.azure.elastic-cloud.com/config/v1/agents?service.name=MyApp&service.environment=production', Version: 1.1, Content: , Headers: { User-Agent: apm-agent-dotnet/1.14.0+a04553cf41c2bc212d866c6031f087273edb9a8b User-Agent: (MyApp 1.0.0) User-Agent: System.Net.Http/5.0.1422.5710 User-Agent: .NET_5/5.0.14 Authorization: [REDACTED] } +-> Response: StatusCode: 401, ReasonPhrase: 'Unauthorized', Version: 1.1, Content: System.Net.Http.HttpConnectionResponseContent, Headers: { Date: Sat, 19 Feb 2022 12:51:49 GMT X-Cloud-Request-Id: iGoZlzp1SZySKJXTKPo3dw X-Content-Type-Options: nosniff X-Found-Handling-Cluster: 575496a0a6cc4dd088c1158558776003 X-Found-Handling-Instance: instance-0000000000 Content-Length: 139 Content-Type: text/plain; charset=utf-8 } +-> Response body [length: 139]: {"error":"authentication failed: improperly encoded ApiKey credentials: expected base64(ID:APIKey): illegal base64 data at input byte 16"}

This is a sample token i have used to test (now revoked), KScUc3zsCPbt5er3H7.
somehow every base64 validator i used is saying this is invalid base64 token, but the example shown in APM Integration page uses the token in json config directly.

i tried to base64 encode the secret, and still got the same error.

Steps to reproduce:
1.
2.
3.

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

Elastic.Apm: Error: {CentralConfigurationFetcher} Exception was thrown while fetching configuration from APM Server and parsing it. ETag: <null>'. URL: https://575496a0a6cc4dd088c1158558776003.apm.southeastasia.azure.elastic-cloud.com/config/v1/agents?service.name=MyApp&service.environment=production'. Apm Server base URL: `https://575496a0a6cc4dd088c1158558776003.apm.southeastasia.azure.elastic-cloud.com/'. WaitInterval: 5m. dbgIterationsCount: 1.
+-> Request:
Method: GET, RequestUri: 'https://575496a0a6cc4dd088c1158558776003.apm.southeastasia.azure.elastic-cloud.com/config/v1/agents?service.name=MyApp&service.environment=production', Version: 1.1, Content: , Headers:
{
User-Agent: apm-agent-dotnet/1.14.0+a04553cf41c2bc212d866c6031f087273edb9a8b
User-Agent: (MyApp 1.0.0)
User-Agent: System.Net.Http/5.0.1422.5710
User-Agent: .NET_5/5.0.14
Authorization: [REDACTED]
}
+-> Response:
StatusCode: 401, ReasonPhrase: 'Unauthorized', Version: 1.1, Content: System.Net.Http.HttpConnectionResponseContent, Headers:
{
Date: Sat, 19 Feb 2022 13:05:25 GMT
X-Cloud-Request-Id: 7HLQTRrCTLW5seFhmtQVHw
X-Content-Type-Options: nosniff
X-Found-Handling-Cluster: 575496a0a6cc4dd088c1158558776003
X-Found-Handling-Instance: instance-0000000000
Content-Length: 139
Content-Type: text/plain; charset=utf-8
}
+-> Response body [length: 139]:
{"error":"authentication failed: improperly encoded ApiKey credentials: expected base64(ID:APIKey): illegal base64 data at input byte 16"}

Hi @inbox.ex ,

tldr: you can ignore this error in the log.

There was an entry about this earlier, you can find it here, which includes more details: .NET An error occurred while sending the request - #2 by pancar

awesome. thanks.

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.