Kibana version:
7.13.2
Elasticsearch version:
7.13.2
APM Server version:
7.13.2
APM Agent language and version:
N/A
Browser version:
N/A
Original install method (e.g. download page, yum, deb, from source, etc.) and version:
ECK (1.6.0)
Fresh install or upgraded from other version?
Fresh Install
Is there anything special in your setup?
No
Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
When trying to create and test an API key to use for APM, I consistently get an unauthorized error.
Steps to reproduce:
- Deploy APM server via ECK:
---
apiVersion: apm.k8s.elastic.co/v1
kind: ApmServer
metadata:
name: apm-prod
namespace: apm-prod
spec:
version: 7.13.2
count: 1
elasticsearchRef:
name: es-prod
namespace: elastic-prod
kibanaRef:
name: kibana-prod
namespace: kibana-prod
http:
tls:
certificate:
secretName: apm-cert
config:
apm-server:
auth:
api_key:
enabled: true
limit: 100
capture_personal_data: true # TODO check if we want this enabled
# TODO RUM?
kibana:
enabled: true
ssl.enabled: true
ssl.verification_mode: certificate
ssl.certificate_authorities: ["/usr/share/apm-server/certs/<snipped>"]
output:
elasticsearch:
ssl.enabled: true
ssl.certificate_authorities: ["/usr/share/apm-server/certs/<snipped>"]
ssl.verification_mode: certificate
http:
enabled: true
host: 0.0.0.0
port: 5067
monitoring:
enabled: false
cluster_uuid: "<snipped>"
podTemplate:
metadata:
annotations:
linkerd.io/inject: enabled
config.linkerd.io/proxy-cpu-limit: "2"
co.elastic.metrics/raw: '[{"enabled":true,"module":"beat","hosts":["http://${data.host}:5067"],"metricsets":["stats","state"],"period":"10s","timeout":"3s","xpack":{"enabled":true}}]'
spec:
automountServiceAccountToken: true
containers:
- name: apm-server
resources:
limits:
memory: 2Gi
cpu: 2
volumeMounts:
- name: <snipped>
mountPath: /usr/share/apm-server/certs
volumes:
- name: <snipped>
secret:
secretName: <snipped>
affinity:
podAntiAffinity:
preferredDuringSchedulingIgnoredDuringExecution:
- weight: 100
podAffinityTerm:
labelSelector:
matchLabels:
elasticsearch.k8s.elastic.co/cluster-name: es-prod
topologyKey: kubernetes.io/hostname
- APM Server connects successfully to both Kibana and Elasticsearch:
{"log.level":"info","@timestamp":"2021-07-02T14:55:07.998Z","log.logger":"kibana","log.origin":{"file.name":"kibana/connecting_client.go","file.line":83},"message":"Successfully obtained connection to Kibana.","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2021-07-02T14:55:07.968Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":364},"message":"GET https://es-prod-es-http.elastic-prod.svc:9200/_xpack <nil>","ecs.version":"1.6.0"}
- Create APM Agent API key based off of guide.
POST /_security/api_key
{
"name": "go-agent",
"role_descriptors": {
"apm": {
"applications": [
{
"application": "apm",
"privileges": ["sourcemap:write", "event:write", "config_agent:read"],
"resources": ["*"]
}
]
}
}
}
- Base64 encode API ID:Key
echo -n '<ID>:<Key>' | base64
- Test API Key:
curl -X GET -H "Authorization: ApiKey <base64_encoded_apiKey>" https://<apm_server_url>:8200/intake/v2/events
- Get error:
{
"error": "unauthorized"
}
- Error in APM server logs:
{"log.level":"error","@timestamp":"2021-07-02T15:19:06.993Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":60},"message":"unauthorized","url.original":"/intake/v2/events","http.request.method":"GET","user_agent.original":"curl/7.66.0","source.address":"127.0.0.1","http.request.body.bytes":0,"http.request.id":"babb3ec6-acf8-4c11-96c3-4b348b7a087c","event.duration":138858,"http.response.status_code":401,"error.message":"unauthorized","ecs.version":"1.6.0"}
Side note: attempting to run the documented command:
curl -H "Authorization: ApiKey <base64_encoded_apiKey>" https://<elasticsearch_server_url>:9200/_security/_authentication
Results in the following error:
{"error":"Incorrect HTTP method for uri [/_security/_authentication] and method [GET], allowed: [POST]","status":405}
Attempting to use POST
curl -X POST -H "Authorization: ApiKey <base64_encoded_apiKey>" https://<elasticsearch_server_url>:9200/_security/_authentication
Results in error:
{"error":{"root_cause":[{"type":"parse_exception","reason":"request body is required"}],"type":"parse_exception","reason":"request body is required"},"status":400}