APM java agent and api_key

GershonA · August 17, 2020, 1:59pm

Hello All,
I'm using elastic cloud service (7.8) and want to allow

elastic-apm-agent-1.18.0.RC1

to use API key instead of secret_token
So, I create the key as following:

POST /_security/api_key
    {
      "name": "my-api-key",
      "expiration": "1d", 
      "role_descriptors": { 
        "apm-privileges": {
          "cluster": ["all"],
          "index": [
            {
              "names": ["apm*"],
              "privileges": ["all"]
            }
          ]
        }
      }
    }

Then I create a role:

PUT /_security/role/apm-privileges 
{
	"applications": [{
	  "application": "apm",
	  "privileges": ["sourcemap:write", "event:write", "config_agent:read"],
	  "resources": ["*"]
	}]
}

The key was base64 encoded (id:api_key)

echo -n 'XXXXXXX:XXXXXXXX' | base64

Now I try to use it as following:

 java -javaagent:"C:\Users\xxxxx\Downloads\elastic-apm-agent-1.18.0.RC1.jar" \
    -Delastic.apm.service_name=apm \
    -Delastic.apm.server_url=https://XXXXXXXXXXX.apm.us-east-1.aws.cloud.es.io:443 \
    -Delastic.apm.application_packages=org.example \
    -Delastic.apm.elastic.apm.environment=int \
    -Delastic.apm.transaction_sample_rate=1.0 \
    -Delastic.apm.elastic.apm.capture_body=OFF \
    -Delastic.apm.elastic.apm.api_key="XXXXXXXXXXXXXXXX==" \
    -jar target/hello-world-rest-api.jar

Error:

2020-08-17 16:33:43,894 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: cannot retry due to server authentication, in streaming mode, response code is 401

The same work well if I use
-Delastic.apm.secret_token=XXXXXXX
instead of
-Delastic.apm.elastic.apm.api_key="XXXXXXXXXXXXXXXX==
Thank You!

simitt · August 17, 2020, 2:57pm

Hi @GershonA,

please create the role and assign the required privileges to the user requesting to create the API Key before creating the API Key.

When creating the API Key try the following request body for assigning all APM privileges:

POST /_security/api_key
{
    "name": "apm-api-key", 
    "expiration": "1d", 
    "role_descriptors": {
        "apm-privileges": {
            "applications": [
               {"application": "apm", 
                 "privileges": ["sourcemap:write", "event:write", "config_agent:read"], 
                 "resources": ["*"]}
            ]
        }
    }
}

GershonA · August 18, 2020, 7:19am

Thank You for answer.
I did all from scratch, with wide permission range as superuser.
So, the role is:

Then I create the key as following:

    POST /_security/api_key
    {
        "name": "apm-api-key", 
        "expiration": "1d", 
        "role_descriptors": {
            "apm-privileges": {
                "applications": [
                   {"application": "MyApp", 
                     "privileges": ["all"], 
                     "resources": ["*"]}
                ]
            }
        }
    }

Response:

{
  "id" : "ER9lAHQBljG9hC72XUFY",
  "name" : "apm-api-key",
  "expiration" : 1597820877122,
  "api_key" : "RHHkxzJZRM2MuNgDuwGoTQ"
}

Then I encode the id:key

echo -n 'ER9lAHQBljG9hC72XUFY:RHHkxzJZRM2MuNgDuwGoTQ' | base64
RVI5bEFIUUJsakc5aEM3MlhVRlk6UkhIa3h6SlpSTTJNdU5nRHV3R29UUQ==

Trying:

java -javaagent:"C:\Users\XXXXXX\Downloads\elastic-apm-agent-1.18.0.RC1.jar" \
-Delastic.apm.service_name=MyApp \
-Delastic.apm.server_url=https://XXXXXXXXXX.apm.us-east-1.aws.cloud.es.io:443 \
-Delastic.apm.application_packages=org.example \
-Delastic.apm.elastic.apm.environment=int \
-Delastic.apm.transaction_sample_rate=1.0 \
-Delastic.apm.elastic.apm.capture_body=OFF \
-Delastic.apm.elastic.apm.api_key="RVI5bEFIUUJsakc5aEM3MlhVRlk6UkhIa3h6SlpSTTJNdU5nRHV3R29UUQ==" \
-jar target/hello-world-rest-api.jar

Same authentication problem

2020-08-18 10:15:52,091 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: cannot retry due to server authentication, in streaming mode, response code is 401

Where I'm wrong?

Thank You for help!

simitt · August 18, 2020, 11:08am

Hi @GershonA,
I see that you have changed the code snippet I shared - please change application back to apm and privileges to "sourcemap:write", "event:write", "config_agent:read".

Regarding the roles and permissions, if you are sending the request as a superuser you do not need to assign the additional role, otherwise please ensure the user has the previously mentioned role with the APM application API Key privileges assigned.

Do not share any valid API Key tokens in a public forum (and in case it happens ensure to immediately invalidate them).

Let me know if this helps.

GershonA · August 18, 2020, 2:36pm

Thank You for trying help me.
Suggested solution not working for superuser and regular user as well.
Steps:

role created:

PUT /_security/role/apm-privileges {
	"applications": [{
	  "application": "apm",
	  "privileges": ["sourcemap:write", "event:write", "config_agent:read"],
	  "resources": ["*"]
	}]
}

API Key created

POST /_security/api_key
{
    "name": "apm-api-key", 
    "expiration": "1d", 
    "role_descriptors": {
        "apm-privileges": {
            "applications": [
               {"application": "apm", 
                 "privileges": ["sourcemap:write", "event:write", "config_agent:read"], 
                 "resources": ["*"]}
            ]
        }
    }
}

id:api_key encoded

java -javaagent:"C:\Users\xxxxx\Downloads\elastic-apm-agent-1.18.0.RC1.jar" \
    -Delastic.apm.service_name=apm \
    -Delastic.apm.server_url=https://XXXXXXXXXXX.apm.us-east-1.aws.cloud.es.io:443 \
    -Delastic.apm.application_packages=org.example \
    -Delastic.apm.elastic.apm.environment=int \
    -Delastic.apm.transaction_sample_rate=1.0 \
    -Delastic.apm.elastic.apm.capture_body=OFF \
    -Delastic.apm.elastic.apm.api_key="(base 64 encoded id:api_key)" \
    -jar target/hello-world-rest-api.jar

Same 401 error

simitt · August 18, 2020, 3:23pm

When using above listed snippet with a superuser to create the API Key I can succesfully ingest data with it. For locating the issue, would you mind trying a manual request against your APM Server and check whether the request is allowed or not. You can download some testdata.ndjson and make a request like:
curl -i -H "Content-type: application/x-ndjson" -H "Authorization: ApiKey your-base-64-encoded-key" --data-binary @testdata.ndjson <your-apm-server-url>/intake/v2/events

If this returns a 202 it would mean that the issue is probably on the java agent side;
if you get a 401 either the key is invalid or API Keys are not enabled in the APM Server. In this case please verify that

the user creating the API Key has expected privileges by running following query (with this user logged into Kibana), expecting the response to return true for every privilege.

GET /_security/user/_has_privileges
{
  "application": [
    {
      "application": "apm",
      "privileges" : [ "sourcemap:write", "event:write", "config_agent:read" ],
      "resources" : [ "*" ]
    }
  ]
}

check that API Key usage is enabled by ensuring apm-server.api_key.enabled=true (default case in Elastic Cloud, if you have not overwritten it in the user settings section it is set to true).

GershonA · August 19, 2020, 7:40am

Hello,
I did as suggested.

With superuser i get return 202
With the apm user 202 as well
From user Kubana:

{
  "username" : "apm_test_user",
  "has_all_requested" : true,
  "cluster" : { },
  "index" : { },
  "application" : {
    "apm" : {
      "*" : {
        "sourcemap:write" : true,
        "event:write" : true,
        "config_agent:read" : true
      }
    }
  }
}

From application:

2020-08-19 10:04:28,303 [main] INFO  co.elastic.apm.agent.util.JmxUtils - Found JVM-specific OperatingSystemMXBean interface: com.sun.management.OperatingSystemMXBean
2020-08-19 10:04:28,558 [main] INFO  co.elastic.apm.agent.configuration.StartupInfo - Starting Elastic APM 1.18.0.RC1 as apm on Java 11.0.8 Runtime version: 11.0.8+10-LTS VM version: 11.0.8+10-LTS (Oracle Corporation) Windows 
10 10.0
2020-08-19 10:04:31,408 [main] INFO  co.elastic.apm.agent.impl.ElasticApmTracer - Tracer switched to RUNNING state
2020-08-19 10:04:31,836 [elastic-apm-server-healthcheck] INFO  co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server is available: 
2020-08-19 10:04:31,840 [elastic-apm-server-healthcheck] WARN  co.elastic.apm.agent.report.ApmServerHealthChecker - Failed to parse version of APM server https://xxx.apm.us-east-1.aws.cloud.es.io:443/: Unexpected end of JSON inpu

Sylvain_Juge · August 19, 2020, 9:41am

Hi @GershonA,

This line in the logs seems to indicate that there is an issue communicating with server, could you provide us debug logs with log_level=debug (see logging troubleshooting doc for reference )

GershonA · August 19, 2020, 10:03am

Hello,
I added following line:
-Delastic.apm.elastic.apm.log_level=DEBUG
The error was not changed

2020-08-19 12:55:02,295 [main] INFO  co.elastic.apm.agent.util.JmxUtils - Found JVM-specific OperatingSystemMXBean interface: com.sun.management.OperatingSystemMXBean
2020-08-19 12:55:02,494 [main] INFO  co.elastic.apm.agent.configuration.StartupInfo - Starting Elastic APM 1.18.0.RC1 as apm on Java 11.0.8 Runtime version: 11.0.8+10-LTS VM version: 11.0.8+10-LTS (Oracle Corporation) Windows 
10 10.0
2020-08-19 12:55:02,506 [main] WARN  co.elastic.apm.agent.configuration.StartupInfo - To enable all features and decrease startup time, please configure application_packages
2020-08-19 12:55:04,557 [main] INFO  co.elastic.apm.agent.impl.ElasticApmTracer - Tracer switched to RUNNING state
2020-08-19 12:55:05,545 [elastic-apm-server-healthcheck] INFO  co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server is available:
2020-08-19 12:55:05,549 [elastic-apm-server-healthcheck] WARN  co.elastic.apm.agent.report.ApmServerHealthChecker - Failed to parse version of APM server https://xxx.apm.us-east-1.aws.cloud.es.io:443/: Unexpected end of JSON input
2020-08-19 12:55:05,550 [elastic-apm-remote-config-poller] ERROR co.elastic.apm.agent.configuration.ApmServerConfigurationSource - Unexpected status 401 while fetching configuration

Sylvain_Juge · August 21, 2020, 8:26am

With a 401 status code, it means there is likely an issue with the api key value, thus just to be sure could you check the agent configuration option value ?

When you generate an API key, you will get a result similar to:

{
  "id" : "XXXXXXXXXXXXXXXXXX",
  "name" : "apm-backend",
  "api_key" : "YYYYYYYYYYYYYYY"
}

You seem to be using windows, but with Linux/bash the instructions to compute the base64 value for API key is the following:

echo -n 'XXXXXXXXXXXXXXXXXX:YYYYYYYYYYYYYYY' | base64
# this will return WFhYWFhYWFhYWFhYWFhYWFhYOllZWVlZWVlZWVlZWVlZWQ==

Here the -n option is really important otherwise an EOL character is included.

GershonA · August 24, 2020, 8:26am

Hello Sylvain_Juge,
I try to run this from windows WSL.
And my ked encoded as suggested:

echo -n 'XXXXXXXXXXXXXXXXXX:YYYYYYYYYYYYYYY' | base64

Unfortunately, the same nasty 401
Thank You!

Sylvain_Juge · August 31, 2020, 9:43am

Do you still have the issue ?

Your server url was reported in the logs (I've since redacted it), and trying to reach it with curl triggered a 502 error, thus I wasn't able to test it myself.

system · September 21, 2020, 5:43am

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.