Authentication failed: missing or improperly formatted Authorization header: expected 'Authorization: Bearer secret_token' or 'Authorization: ApiKey base64(API key ID:API key)'

Elastic Observability
1

What does this error mean?

[my-apm-server-7f75cffcf7-grj2t apm-server] {"log.level":"error","@timestamp":"2023-08-08T05:32:12.641Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":58},"message":"authentication failed: missing or improperly formatted Authorization header: expected 'Authorization: Bearer secret_token' or 'Authorization: ApiKey base64(API key ID:API key)'","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-dotnet/1.22.0+1248195c387cac21578a0da4ff22835d4fd5f1d4 (orders-apm 1.0) System.Net.Http/7.0.523.17405 .NET_7/7.0.5","source.address":"10.0.29.219","http.request.id":"5dfb2feb-3328-4af6-9b3c-b0d0b12ffb8b","event.duration":68170,"http.response.status_code":401,"error.message":"authentication failed: missing or improperly formatted Authorization header: expected 'Authorization: Bearer secret_token' or 'Authorization: ApiKey base64(API key ID:API key)'","ecs.version":"1.6.0"}

[my-apm-server-7f75cffcf7-bmdm5 apm-server] {"log.level":"error","@timestamp":"2023-08-08T05:35:10.283Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":58},"message":"authentication failed: missing or improperly formatted Authorization header: expected 'Authorization: Bearer secret_token' or 'Authorization: ApiKey base64(API key ID:API key)'","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-nodejs/3.45.0 (buyerportal-apm 2.0.0)","source.address":"10.0.93.34","http.request.id":"e43db8cb-93b1-46e1-9851-522268ec10d0","event.duration":89766,"http.response.status_code":401,"error.message":"authentication failed: missing or improperly formatted Authorization header: expected 'Authorization: Bearer secret_token' or 'Authorization: ApiKey base64(API key ID:API key)'","ecs.version":"1.6.0"}

And what should I do?

8.9.0

2

The error message suggests that the agents are not sending an Authorization HTTP header, or that it's improperly formatted. I expect it's the former.

Please see Secure communication with APM agents | APM User Guide [8.9] | Elastic, which has links for how to configure agent authentication.

3

Which option is easier to manage from k8s manifests?

4

Do I have to generate a self-signed TLS cert or is there any ready-to-use cert, for example the <foo>-es-http-ca-internal ca cert?

5

Are you using Elastic Cloud for Kubernetes (ECK)? Based on your last question, I guess so.

I think the simplest thing would be to use the ApmServer kind, and the auto-generated secret token: Connect to the APM Server | Elastic Cloud on Kubernetes [2.9] | Elastic

ECK will auto-generate a self-signed TLS certificate for APM Server by default. This is stored in a Secret called <name>-apm-http-certs-internal.

6

I retrieve the tls cert and create a k8s secret:

kubectl get secret my-apm-http-ca-internal -o go-template='{{index .data "tls.crt" | base64decode }}' > elasticsearch-apm-ca.crt
kubectl create secret generic elasticsearch-eck-apm-ca --from-file=tls.crt=elasticsearch-apm-ca.crt

How do I configure the APM integration TLS?

image
image1548×739 37.7 KB

7

Sorry for dropping the ball on this topic - I got busy and forgot about it.

If you're using the ApmServer kind, you don't need to configure the APM integration. The APM integration is only needed if you intend to run APM Server with Elastic Agent. When you're using the ApmServer kind, ECK is setting up the apm-server binary similar to what's shown in APM Server binary | APM User Guide [8.9] | Elastic

Let's keep discussing over in How to get ECK APM to work with any type of agent at all!?! - #8 by axw