:8200/intake/v2/events
```
```auto
{
"error": "unauthorized"
}
```
---
### Author:  [@BenB196](https://discuss.elastic.co/u/BenB196)
#### Post date: [July 5, 2021, 2:49pm UTC](https://discuss.elastic.co/t/apm-apikey-failing-with-unauthorized-wrong-permissions-documented/277663/4 "2021-07-05T14:49:51Z")
@axw I believe I found the issue, when I tried to generate the token via the apm-server itself:
```auto
apm-server -c config/config-secret/apm-server.yml apikey create --ingest --agent-config --name go-app
```
I got the following error:
```auto
apm-prod-apm-prod-apm-user is missing the following requested privilege(s): event:write, config_agent:read.
You might try with the superuser, or add the APM application privileges to the role of the authenticated user, eg.:
PUT /_security/role/my_role {
...
"applications": [{
"application": "apm",
"privileges": ["sourcemap:write", "event:write", "config_agent:read"],
"resources": ["*"]
}],
...
}
```
It looks like the user that was generated by ECK for this deployment doesn't have the correct permissions.
I'm not sure if this is an issue with ECK or with APM, would you have any more insight into this?
---
### Author:  [@BenB196](https://discuss.elastic.co/u/BenB196)
#### Post date: [July 5, 2021, 3:00pm UTC](https://discuss.elastic.co/t/apm-apikey-failing-with-unauthorized-wrong-permissions-documented/277663/5 "2021-07-05T15:00:56Z")
Looking into this a bit more, it looks like ECK adds the user to the Elasticsearch deployment with the following roles:
```auto
eck_apm_user_role_v75,ingest_admin,apm_system
```
Not sure if these roles cover everything.
But one thing I did notice, is that if I try to get the user:
```auto
GET /_security/user/apm-prod-apm-prod-apm-user
```
I get a 404:
```auto
{ }
```
Not sure if this is intended, or if ECK never properly added the APM user to the cluster.
---
### Author:  [@axw](https://discuss.elastic.co/u/axw)
#### Post date: [July 7, 2021, 3:43am UTC](https://discuss.elastic.co/t/apm-apikey-failing-with-unauthorized-wrong-permissions-documented/277663/6 "2021-07-07T03:43:34Z")
> It looks like the user that was generated by ECK for this deployment doesn't have the correct permissions.
Right, the built-in `apm_user` role does not have sufficient privileges for creating API Keys. This is by design, to keep the privileges minimal.
The simplest way to create an API Key would be to use the `elastic` superuser. I think in your case, the secret would be `elastic-prod-elastic-user`. Alternatively, you can follow [Grant privileges and roles needed for API key management | APM User Guide [8.11] | Elastic](https://www.elastic.co/guide/en/apm/server/current/privileges-api-key.html) to create a more targeted role for creating the API Keys.
FYI we will be introducing a UI to manage of APM agent API Keys in the future, which I expect will simplify this: [[APM] Agent API Key Management · Issue #77966 · elastic/kibana · GitHub](https://github.com/elastic/kibana/issues/77966)
---
### Author:  [@BenB196](https://discuss.elastic.co/u/BenB196)
#### Post date: [July 7, 2021, 5:40pm UTC](https://discuss.elastic.co/t/apm-apikey-failing-with-unauthorized-wrong-permissions-documented/277663/7 "2021-07-07T17:40:40Z")
@axw so I did some more testing, but am still seeing an issue:
I created a new API Key specifically for the APM Server via (using the elastic superadmin account), and deployed it to the APM Server:
```json
POST /_security/api_key
{
"name": "apm_server_api_key",
"role_descriptors": {
"apm_setup": {
"cluster": [
"manage_ilm", "manage_index_templates", "manage_pipeline"
],
"index": [
{
"names": ["apm-*"],
"privileges": ["manage"]
}
]
},
"apm_writer": {
"index": [
{
"names": ["apm-*"],
"privileges": ["create_doc","create_index"]
},
{
"names": ["apm-*sourcemap"],
"privileges": ["read"]
}
]
},
"apm_system": {
"cluster": ["monitor","cluster:admin/xpack/monitoring/bulk"],
"index": [
{
"names": [".monitoring-beats-*"],
"privileges": ["create_index","create_doc"]
}
]
},
"apm_manage_api_key": {
"cluster": [
"manage_api_key"
],
"applications": [
{
"application": "apm",
"privileges": ["sourcemap:write", "event:write", "config_agent:read"],
"resources": ["*"]
}
]
}
}
}
```
The APM server appears to startup and connect to Elasticsearch and Kibana perfectly fine.
However, when I run:
```auto
apm-server -c config/config-secret/apm-server.yml apikey create --ingest --agent-config --name go-app
```
I am seeing the error:
```auto
"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"creating derived api keys requires an explicit role descriptor that is empty (has no privileges)"}],"type":"illegal_argument_exception","reason":"creating derived api keys requires an explicit role descriptor that is empty (has no privileges)"},"status":400}
```
---
### Author:  [@axw](https://discuss.elastic.co/u/axw)
#### Post date: [July 8, 2021, 3:09am UTC](https://discuss.elastic.co/t/apm-apikey-failing-with-unauthorized-wrong-permissions-documented/277663/8 "2021-07-08T03:09:53Z")
Per [Create API key API | Elasticsearch Guide [7.13] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-api-key.html), you cannot use an API Key to create another API Key except if the target API Key is to have no privileges. So I'm afraid that this approach isn't going to work. You would have to instead create a user for apm-server with those roles, and set up apm-server with user/password basic auth.
---
### Author:  [@BenB196](https://discuss.elastic.co/u/BenB196)
#### Post date: [July 8, 2021, 1:46pm UTC](https://discuss.elastic.co/t/apm-apikey-failing-with-unauthorized-wrong-permissions-documented/277663/9 "2021-07-08T13:46:50Z")
@axw thanks I completely missed that part.
I've created a new user with the following:
```auto
{
"apm_server" : {
"username" : "apm_server",
"roles" : [
"apm_system",
"ingest_admin",
"apm_manage_api_key",
"apm_setup",
"apm_writer"
],
"full_name" : "",
"email" : "",
"metadata" : { },
"enabled" : true
}
}
```
apm\_manage\_api\_key role:
```auto
{
"apm_manage_api_key" : {
"cluster" : [
"manage_api_key"
],
"indices" : [],
"applications" : [
{
"application" : "apm",
"privileges" : [
"sourcemap:write",
"event:write",
"config_agent:read"
],
"resources" : [
"*"
]
}
],
"run_as" : [],
"metadata" : { },
"transient_metadata" : {
"enabled" : true
}
}
}
```
apm\_setup role:
```auto
{
"apm_setup" : {
"cluster" : [
"manage_ilm",
"manage_index_templates",
"manage_pipeline"
],
"indices" : [
{
"names" : [
"apm-*"
],
"privileges" : [
"manage"
],
"field_security" : {
"grant" : [
"*"
],
"except" : []
},
"allow_restricted_indices" : false
}
],
"applications" : [],
"run_as" : [],
"metadata" : { },
"transient_metadata" : {
"enabled" : true
}
}
}
```
apm\_writer role:
```auto
{
"apm_writer" : {
"cluster" : [],
"indices" : [
{
"names" : [
"apm-*"
],
"privileges" : [
"create_doc",
"create"
],
"field_security" : {
"grant" : [
"*"
],
"except" : []
},
"allow_restricted_indices" : false
},
{
"names" : [
"apm-*sourcemap"
],
"privileges" : [
"read"
],
"field_security" : {
"grant" : [
"*"
]
},
"allow_restricted_indices" : false
}
],
"applications" : [],
"run_as" : [],
"metadata" : { },
"transient_metadata" : {
"enabled" : true
}
}
}
```
I then generated an API with the apm\_server user:
```auto
POST /_security/api_key
{
"name": "go-agent",
"role_descriptors": {
"apm": {
"applications": [
{
"application": "apm",
"privileges": ["sourcemap:write", "event:write", "config_agent:read"],
"resources": ["*"]
}
]
}
}
}
```
If I run the API Key through the APM Server check everything passes:
```auto
apm-server -c config/config-secret/apm-server.yml apikey verify --credentials
```
```auto
Authorized for privilege "config_agent:read"...: Yes
Authorized for privilege "event:write"...: Yes
Authorized for privilege "sourcemap:write"...: Yes
```
And if I try the key directly against Elasticsearch via:
```auto
curl -H "Authorization: ApiKey " https://:9200/_security/_authenticate
```
I get:
```auto
{"username":"apm_server","roles":[],"full_name":"","email":"","metadata":{},"enabled":true,"authentication_realm":{"name":"_es_api_key","type":"_es_api_key"},"lookup_realm":{"name":"_es_api_key","type":"_es_api_key"},"authentication_type":"api_key"}
```
But when I try to execute the curl command against the APM server via:
```auto
curl --cacert /etc/pki/trust/anchors/.pem -H "Authorization: ApiKey " https://:8200/intake/v2/events
```
I am still getting an unauthorized:
```auto
{
"error": "unauthorized"
}
```
---
### Author:  [@axw](https://discuss.elastic.co/u/axw)
#### Post date: [July 9, 2021, 5:52am UTC](https://discuss.elastic.co/t/apm-apikey-failing-with-unauthorized-wrong-permissions-documented/277663/10 "2021-07-09T05:52:49Z")
I overlooked something in your server config earlier:
```auto
apm-server:
auth:
api_key:
enabled: true
limit: 100
```
I guess you were referring to the `apm-server.yml` in github, on the master branch? There have been some recent changes that were not yet released in 7.13.2. Try changing this to:
```auto
apm-server:
api_key:
enabled: true
limit: 100
```
We are reorganising some of the auth-related config under `apm-server.auth`, including `api_key` and `secret_token`. The old name for `apm-server.auth.api_key` config is `apm-server.api_key`.
---
### Author:  [@BenB196](https://discuss.elastic.co/u/BenB196)
#### Post date: [July 9, 2021, 12:31pm UTC](https://discuss.elastic.co/t/apm-apikey-failing-with-unauthorized-wrong-permissions-documented/277663/11 "2021-07-09T12:31:57Z")
@axw yep, that was the issue. I wasn't able to find the default config on the docs page so went to GitHub, and didn't think about changing the branch.
Thanks for the help on this.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 30, 2021, 8:32am UTC](https://discuss.elastic.co/t/apm-apikey-failing-with-unauthorized-wrong-permissions-documented/277663/12 "2021-07-30T08:32:43Z")
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.