APM ApiKey Failing with Unauthorized (Wrong permissions documented?)

Kibana version:
7.13.2

Elasticsearch version:
7.13.2

APM Server version:
7.13.2

APM Agent language and version:
N/A

Browser version:
N/A

Original install method (e.g. download page, yum, deb, from source, etc.) and version:
ECK (1.6.0)

Fresh install or upgraded from other version?
Fresh Install

Is there anything special in your setup?
No

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
When trying to create and test an API key to use for APM, I consistently get an unauthorized error.

Steps to reproduce:

  1. Deploy APM server via ECK:
---
apiVersion: apm.k8s.elastic.co/v1
kind: ApmServer
metadata:
  name: apm-prod
  namespace: apm-prod
spec:
  version: 7.13.2
  count: 1
  elasticsearchRef:
    name: es-prod
    namespace: elastic-prod
  kibanaRef:
    name: kibana-prod
    namespace: kibana-prod
  http:
    tls:
      certificate:
        secretName: apm-cert
  config:
    apm-server:
      auth:
        api_key:
          enabled: true
          limit: 100
      capture_personal_data: true # TODO check if we want this enabled
      # TODO RUM?
      kibana:
        enabled: true
        ssl.enabled: true
        ssl.verification_mode: certificate
        ssl.certificate_authorities: ["/usr/share/apm-server/certs/<snipped>"]
    output:
      elasticsearch:
        ssl.enabled: true
        ssl.certificate_authorities: ["/usr/share/apm-server/certs/<snipped>"]
        ssl.verification_mode: certificate
    http:
      enabled: true
      host: 0.0.0.0
      port: 5067
    monitoring:
      enabled: false
      cluster_uuid: "<snipped>"
  podTemplate:
    metadata:
      annotations:
        linkerd.io/inject: enabled
        config.linkerd.io/proxy-cpu-limit: "2"
        co.elastic.metrics/raw: '[{"enabled":true,"module":"beat","hosts":["http://${data.host}:5067"],"metricsets":["stats","state"],"period":"10s","timeout":"3s","xpack":{"enabled":true}}]'
    spec:
      automountServiceAccountToken: true
      containers:
        - name: apm-server
          resources:
            limits:
              memory: 2Gi
              cpu: 2
          volumeMounts:
            - name: <snipped>
              mountPath: /usr/share/apm-server/certs
      volumes:
        - name: <snipped>
          secret:
            secretName: <snipped>
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchLabels:
                    elasticsearch.k8s.elastic.co/cluster-name: es-prod
                topologyKey: kubernetes.io/hostname
  1. APM Server connects successfully to both Kibana and Elasticsearch:
{"log.level":"info","@timestamp":"2021-07-02T14:55:07.998Z","log.logger":"kibana","log.origin":{"file.name":"kibana/connecting_client.go","file.line":83},"message":"Successfully obtained connection to Kibana.","ecs.version":"1.6.0"}
{"log.level":"debug","@timestamp":"2021-07-02T14:55:07.968Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":364},"message":"GET https://es-prod-es-http.elastic-prod.svc:9200/_xpack <nil>","ecs.version":"1.6.0"}
  1. Create APM Agent API key based off of guide.
POST /_security/api_key
{
  "name": "go-agent",
  "role_descriptors": {
    "apm": {
      "applications": [
        {
          "application": "apm",
          "privileges": ["sourcemap:write", "event:write", "config_agent:read"],
          "resources": ["*"]
        }
      ]
    }
  }
}
  1. Base64 encode API ID:Key
echo -n '<ID>:<Key>' | base64
  1. Test API Key:
curl -X GET -H "Authorization: ApiKey <base64_encoded_apiKey>" https://<apm_server_url>:8200/intake/v2/events
  1. Get error:
{
  "error": "unauthorized"
}
  1. Error in APM server logs:
{"log.level":"error","@timestamp":"2021-07-02T15:19:06.993Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":60},"message":"unauthorized","url.original":"/intake/v2/events","http.request.method":"GET","user_agent.original":"curl/7.66.0","source.address":"127.0.0.1","http.request.body.bytes":0,"http.request.id":"babb3ec6-acf8-4c11-96c3-4b348b7a087c","event.duration":138858,"http.response.status_code":401,"error.message":"unauthorized","ecs.version":"1.6.0"}


Side note: attempting to run the documented command:

curl -H "Authorization: ApiKey <base64_encoded_apiKey>" https://<elasticsearch_server_url>:9200/_security/_authentication

Results in the following error:

{"error":"Incorrect HTTP method for uri [/_security/_authentication] and method [GET], allowed: [POST]","status":405}

Attempting to use POST

curl -X POST -H "Authorization: ApiKey <base64_encoded_apiKey>" https://<elasticsearch_server_url>:9200/_security/_authentication

Results in error:

{"error":{"root_cause":[{"type":"parse_exception","reason":"request body is required"}],"type":"parse_exception","reason":"request body is required"},"status":400}

Sorry, there's a typo in the docs. I've opened docs: fix typo by axw · Pull Request #5600 · elastic/apm-server · GitHub to fix that. The correct URL path is /_security/_authenticate.

Aside from that, I'm not sure where things are going wrong for you. I'll show what worked for me just now.

Create the API Key:

POST /_security/api_key
{
  "name": "go-agent",
  "role_descriptors": {
    "apm": {
      "applications": [
        {
          "application": "apm",
          "privileges": ["sourcemap:write", "event:write", "config_agent:read"],
          "resources": ["*"]
        }
      ]
    }
  }
}

Elasticsearch response:

{
  "id" : "GyZWdHoBbmV-n6Ll1ET0",
  "name" : "go-agent",
  "api_key" : "asEpXFz7Qza3zW9zkVBkaA"
}

Encode credentials:

$ echo -n 'GyZWdHoBbmV-n6Ll1ET0:asEpXFz7Qza3zW9zkVBkaA' | base64
R3laV2RIb0JibVYtbjZMbDFFVDA6YXNFcFhGejdRemEzelc5emtWQmthQQ==

Test credentials with Elasticsearch:

$ curl -H "Authorization: ApiKey R3laV2RIb0JibVYtbjZMbDFFVDA6YXNFcFhGejdRemEzelc5emtWQmthQQ==" http://localhost:9200/_security/_authenticate
{"username":"admin","roles":[],"full_name":null,"email":null,"metadata":{},"enabled":true,"authentication_realm":{"name":"_es_api_key","type":"_es_api_key"},"lookup_realm":{"name":"_es_api_key","type":"_es_api_key"},"authentication_type":"api_key"}

Test credentials with APM Server:

$ curl -H "Authorization: ApiKey R3laV2RIb0JibVYtbjZMbDFFVDA6YXNFcFhGejdRemEzelc5emtWQmthQQ==" http://localhost:8200/intake/v2/events
{                                                  
  "accepted": 0,                                   
  "errors": [                                      
    {                                              
      "message": "only POST requests are supported"
    }                                              
  ]                                                
}                                                  

(That error is expected, and will only be returned if the API Key is verified.)

@axw Thanks for the updated command, after running the new command:

curl -H "Authorization: ApiKey <base64_encoded_apiKey>" https://<elasticsearch_server_url>:9200/_security/_authenticate

I get:

{"username":"<username>","roles":[],"full_name":null,"email":null,"metadata":{"saml_nameid_format":"urn:oasis:names:tc:SAML:1.1:nameid-format:unspecified","saml_nameid":"<saml_username>","saml(groups)":[<saml_groups>]},"enabled":true,"authentication_realm":{"name":"_es_api_key","type":"_es_api_key"},"lookup_realm":{"name":"_es_api_key","type":"_es_api_key"},"authentication_type":"api_key"}

However, I'm still getting an unauthorized error when running:

curl -k -H "Authorization: ApiKey <base64_encoded_apiKey>" https://<apm_server_url>:8200/intake/v2/events
{
  "error": "unauthorized"
}

@axw I believe I found the issue, when I tried to generate the token via the apm-server itself:

apm-server -c config/config-secret/apm-server.yml apikey create --ingest --agent-config --name go-app

I got the following error:

apm-prod-apm-prod-apm-user is missing the following requested privilege(s): event:write, config_agent:read.

You might try with the superuser, or add the APM application privileges to the role of the authenticated user, eg.:
PUT /_security/role/my_role {
        ...
        "applications": [{
          "application": "apm",
          "privileges": ["sourcemap:write", "event:write", "config_agent:read"],
          "resources": ["*"]
        }],
        ...
}

It looks like the user that was generated by ECK for this deployment doesn't have the correct permissions.

I'm not sure if this is an issue with ECK or with APM, would you have any more insight into this?

Looking into this a bit more, it looks like ECK adds the user to the Elasticsearch deployment with the following roles:

eck_apm_user_role_v75,ingest_admin,apm_system

Not sure if these roles cover everything.

But one thing I did notice, is that if I try to get the user:

GET /_security/user/apm-prod-apm-prod-apm-user

I get a 404:

{ }

Not sure if this is intended, or if ECK never properly added the APM user to the cluster.

It looks like the user that was generated by ECK for this deployment doesn't have the correct permissions.

Right, the built-in apm_user role does not have sufficient privileges for creating API Keys. This is by design, to keep the privileges minimal.

The simplest way to create an API Key would be to use the elastic superuser. I think in your case, the secret would be elastic-prod-elastic-user. Alternatively, you can follow Grant privileges and roles needed for API key management | APM Server Reference [7.13] | Elastic to create a more targeted role for creating the API Keys.

FYI we will be introducing a UI to manage of APM agent API Keys in the future, which I expect will simplify this: [APM] Agent API Key Management · Issue #77966 · elastic/kibana · GitHub

@axw so I did some more testing, but am still seeing an issue:

I created a new API Key specifically for the APM Server via (using the elastic superadmin account), and deployed it to the APM Server:

POST /_security/api_key
{
  "name": "apm_server_api_key",
  "role_descriptors": {
    "apm_setup": {
      "cluster": [
        "manage_ilm", "manage_index_templates", "manage_pipeline"
      ],
      "index": [
        {
          "names": ["apm-*"],
          "privileges": ["manage"]
        }
      ]
    },
    "apm_writer": {
      "index": [
        {
          "names": ["apm-*"],
          "privileges": ["create_doc","create_index"]
        },
        {
          "names": ["apm-*sourcemap"],
          "privileges": ["read"]
        }
      ]
    },
    "apm_system": {
      "cluster": ["monitor","cluster:admin/xpack/monitoring/bulk"],
      "index": [
        {
          "names": [".monitoring-beats-*"],
          "privileges": ["create_index","create_doc"]
        }
      ]
    },
    "apm_manage_api_key": {
      "cluster": [
        "manage_api_key"
      ],
      "applications": [
        {
          "application": "apm",
          "privileges": ["sourcemap:write", "event:write", "config_agent:read"],
          "resources": ["*"]
        }
      ]
    }
  }
}

The APM server appears to startup and connect to Elasticsearch and Kibana perfectly fine.

However, when I run:

apm-server -c config/config-secret/apm-server.yml apikey create --ingest --agent-config --name go-app

I am seeing the error:

"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"creating derived api keys requires an explicit role descriptor that is empty (has no privileges)"}],"type":"illegal_argument_exception","reason":"creating derived api keys requires an explicit role descriptor that is empty (has no privileges)"},"status":400}

Per Create API key API | Elasticsearch Guide [7.13] | Elastic, you cannot use an API Key to create another API Key except if the target API Key is to have no privileges. So I'm afraid that this approach isn't going to work. You would have to instead create a user for apm-server with those roles, and set up apm-server with user/password basic auth.

@axw thanks I completely missed that part.

I've created a new user with the following:

{
  "apm_server" : {
    "username" : "apm_server",
    "roles" : [
      "apm_system",
      "ingest_admin",
      "apm_manage_api_key",
      "apm_setup",
      "apm_writer"
    ],
    "full_name" : "",
    "email" : "",
    "metadata" : { },
    "enabled" : true
  }
}

apm_manage_api_key role:

{
  "apm_manage_api_key" : {
    "cluster" : [
      "manage_api_key"
    ],
    "indices" : [ ],
    "applications" : [
      {
        "application" : "apm",
        "privileges" : [
          "sourcemap:write",
          "event:write",
          "config_agent:read"
        ],
        "resources" : [
          "*"
        ]
      }
    ],
    "run_as" : [ ],
    "metadata" : { },
    "transient_metadata" : {
      "enabled" : true
    }
  }
}

apm_setup role:

{
  "apm_setup" : {
    "cluster" : [
      "manage_ilm",
      "manage_index_templates",
      "manage_pipeline"
    ],
    "indices" : [
      {
        "names" : [
          "apm-*"
        ],
        "privileges" : [
          "manage"
        ],
        "field_security" : {
          "grant" : [
            "*"
          ],
          "except" : [ ]
        },
        "allow_restricted_indices" : false
      }
    ],
    "applications" : [ ],
    "run_as" : [ ],
    "metadata" : { },
    "transient_metadata" : {
      "enabled" : true
    }
  }
}

apm_writer role:

{
  "apm_writer" : {
    "cluster" : [ ],
    "indices" : [
      {
        "names" : [
          "apm-*"
        ],
        "privileges" : [
          "create_doc",
          "create"
        ],
        "field_security" : {
          "grant" : [
            "*"
          ],
          "except" : [ ]
        },
        "allow_restricted_indices" : false
      },
      {
        "names" : [
          "apm-*sourcemap"
        ],
        "privileges" : [
          "read"
        ],
        "field_security" : {
          "grant" : [
            "*"
          ]
        },
        "allow_restricted_indices" : false
      }
    ],
    "applications" : [ ],
    "run_as" : [ ],
    "metadata" : { },
    "transient_metadata" : {
      "enabled" : true
    }
  }
}

I then generated an API with the apm_server user:

POST /_security/api_key
{
  "name": "go-agent",
  "role_descriptors": {
    "apm": {
      "applications": [
        {
          "application": "apm",
          "privileges": ["sourcemap:write", "event:write", "config_agent:read"],
          "resources": ["*"]
        }
      ]
    }
  }
}

If I run the API Key through the APM Server check everything passes:

apm-server -c config/config-secret/apm-server.yml apikey verify --credentials <base64_api_id:key>
Authorized for privilege "config_agent:read"...: Yes
Authorized for privilege "event:write"...: Yes
Authorized for privilege "sourcemap:write"...: Yes

And if I try the key directly against Elasticsearch via:

curl -H "Authorization: ApiKey <base64_api_id:key>" https://<elasticsearch_url>:9200/_security/_authenticate

I get:

{"username":"apm_server","roles":[],"full_name":"","email":"","metadata":{},"enabled":true,"authentication_realm":{"name":"_es_api_key","type":"_es_api_key"},"lookup_realm":{"name":"_es_api_key","type":"_es_api_key"},"authentication_type":"api_key"}

But when I try to execute the curl command against the APM server via:

curl --cacert /etc/pki/trust/anchors/<root_ca>.pem -H "Authorization: ApiKey <base64_api_id:key>" https://<apm_server_url>:8200/intake/v2/events

I am still getting an unauthorized:

{
  "error": "unauthorized"
}

I overlooked something in your server config earlier:

apm-server:
  auth:
    api_key:
      enabled: true
      limit: 100

I guess you were referring to the apm-server.yml in github, on the master branch? There have been some recent changes that were not yet released in 7.13.2. Try changing this to:

apm-server:
  api_key:
    enabled: true
    limit: 100

We are reorganising some of the auth-related config under apm-server.auth, including api_key and secret_token. The old name for apm-server.auth.api_key config is apm-server.api_key.

@axw yep, that was the issue. I wasn't able to find the default config on the docs page so went to GitHub, and didn't think about changing the branch.

Thanks for the help on this.

1 Like

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.