khteh
August 8, 2023, 5:34am
1
What does this error mean?
[my-apm-server-7f75cffcf7-grj2t apm-server] {"log.level":"error","@timestamp":"2023-08-08T05:32:12.641Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":58},"message":"authentication failed: missing or improperly formatted Authorization header: expected 'Authorization: Bearer secret_token' or 'Authorization: ApiKey base64(API key ID:API key)'","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-dotnet/1.22.0+1248195c387cac21578a0da4ff22835d4fd5f1d4 (orders-apm 1.0) System.Net.Http/7.0.523.17405 .NET_7/7.0.5","source.address":"10.0.29.219","http.request.id":"5dfb2feb-3328-4af6-9b3c-b0d0b12ffb8b","event.duration":68170,"http.response.status_code":401,"error.message":"authentication failed: missing or improperly formatted Authorization header: expected 'Authorization: Bearer secret_token' or 'Authorization: ApiKey base64(API key ID:API key)'","ecs.version":"1.6.0"}
[my-apm-server-7f75cffcf7-bmdm5 apm-server] {"log.level":"error","@timestamp":"2023-08-08T05:35:10.283Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":58},"message":"authentication failed: missing or improperly formatted Authorization header: expected 'Authorization: Bearer secret_token' or 'Authorization: ApiKey base64(API key ID:API key)'","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-nodejs/3.45.0 (buyerportal-apm 2.0.0)","source.address":"10.0.93.34","http.request.id":"e43db8cb-93b1-46e1-9851-522268ec10d0","event.duration":89766,"http.response.status_code":401,"error.message":"authentication failed: missing or improperly formatted Authorization header: expected 'Authorization: Bearer secret_token' or 'Authorization: ApiKey base64(API key ID:API key)'","ecs.version":"1.6.0"}
And what should I do?
8.9.0
axw
August 8, 2023, 7:01am
2
The error message suggests that the agents are not sending an Authorization HTTP header, or that it's improperly formatted. I expect it's the former.
Please see Secure communication with APM agents | APM User Guide [8.9] | Elastic , which has links for how to configure agent authentication.
khteh
August 8, 2023, 9:19am
3
Which option is easier to manage from k8s manifests?
khteh
August 8, 2023, 9:21am
4
Do I have to generate a self-signed TLS cert or is there any ready-to-use cert, for example the <foo>-es-http-ca-internal ca cert?
axw
August 9, 2023, 2:21am
5
Are you using Elastic Cloud for Kubernetes (ECK)? Based on your last question, I guess so.
I think the simplest thing would be to use the ApmServer kind, and the auto-generated secret token: Connect to the APM Server | Elastic Cloud on Kubernetes [2.9] | Elastic
ECK will auto-generate a self-signed TLS certificate for APM Server by default. This is stored in a Secret called <name>-apm-http-certs-internal.
khteh
August 9, 2023, 1:06pm
6
I retrieve the tls cert and create a k8s secret:
kubectl get secret my-apm-http-ca-internal -o go-template='{{index .data "tls.crt" | base64decode }}' > elasticsearch-apm-ca.crt
kubectl create secret generic elasticsearch-eck-apm-ca --from-file=tls.crt=elasticsearch-apm-ca.crt
How do I configure the APM integration TLS?
axw
August 29, 2023, 2:40am
7
Sorry for dropping the ball on this topic - I got busy and forgot about it.
If you're using the ApmServer kind, you don't need to configure the APM integration. The APM integration is only needed if you intend to run APM Server with Elastic Agent. When you're using the ApmServer kind, ECK is setting up the apm-server binary similar to what's shown in APM Server binary | APM User Guide [8.9] | Elastic
Let's keep discussing over in How to get ECK APM to work with any type of agent at all!?! - #8 by axw
