2020-08-17 16:33:43,894 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: cannot retry due to server authentication, in streaming mode, response code is 401
The same work well if I use -Delastic.apm.secret_token=XXXXXXX
instead of -Delastic.apm.elastic.apm.api_key="XXXXXXXXXXXXXXXX==
Thank You!
2020-08-18 10:15:52,091 [elastic-apm-server-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: cannot retry due to server authentication, in streaming mode, response code is 401
Hi @GershonA,
I see that you have changed the code snippet I shared - please change application back to apm and privileges to "sourcemap:write", "event:write", "config_agent:read".
Regarding the roles and permissions, if you are sending the request as a superuser you do not need to assign the additional role, otherwise please ensure the user has the previously mentioned role with the APM application API Key privileges assigned.
Do not share any valid API Key tokens in a public forum (and in case it happens ensure to immediately invalidate them).
When using above listed snippet with a superuser to create the API Key I can succesfully ingest data with it. For locating the issue, would you mind trying a manual request against your APM Server and check whether the request is allowed or not. You can download some testdata.ndjson and make a request like: curl -i -H "Content-type: application/x-ndjson" -H "Authorization: ApiKey your-base-64-encoded-key" --data-binary @testdata.ndjson <your-apm-server-url>/intake/v2/events
If this returns a 202 it would mean that the issue is probably on the java agent side;
if you get a 401 either the key is invalid or API Keys are not enabled in the APM Server. In this case please verify that
the user creating the API Key has expected privileges by running following query (with this user logged into Kibana), expecting the response to return true for every privilege.
check that API Key usage is enabled by ensuring apm-server.api_key.enabled=true (default case in Elastic Cloud, if you have not overwritten it in the user settings section it is set to true).
2020-08-19 10:04:28,303 [main] INFO co.elastic.apm.agent.util.JmxUtils - Found JVM-specific OperatingSystemMXBean interface: com.sun.management.OperatingSystemMXBean
2020-08-19 10:04:28,558 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - Starting Elastic APM 1.18.0.RC1 as apm on Java 11.0.8 Runtime version: 11.0.8+10-LTS VM version: 11.0.8+10-LTS (Oracle Corporation) Windows
10 10.0
2020-08-19 10:04:31,408 [main] INFO co.elastic.apm.agent.impl.ElasticApmTracer - Tracer switched to RUNNING state
2020-08-19 10:04:31,836 [elastic-apm-server-healthcheck] INFO co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server is available:
2020-08-19 10:04:31,840 [elastic-apm-server-healthcheck] WARN co.elastic.apm.agent.report.ApmServerHealthChecker - Failed to parse version of APM server https://xxx.apm.us-east-1.aws.cloud.es.io:443/: Unexpected end of JSON inpu
This line in the logs seems to indicate that there is an issue communicating with server, could you provide us debug logs with log_level=debug (see logging troubleshooting doc for reference )
Hello,
I added following line: -Delastic.apm.elastic.apm.log_level=DEBUG
The error was not changed
2020-08-19 12:55:02,295 [main] INFO co.elastic.apm.agent.util.JmxUtils - Found JVM-specific OperatingSystemMXBean interface: com.sun.management.OperatingSystemMXBean
2020-08-19 12:55:02,494 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - Starting Elastic APM 1.18.0.RC1 as apm on Java 11.0.8 Runtime version: 11.0.8+10-LTS VM version: 11.0.8+10-LTS (Oracle Corporation) Windows
10 10.0
2020-08-19 12:55:02,506 [main] WARN co.elastic.apm.agent.configuration.StartupInfo - To enable all features and decrease startup time, please configure application_packages
2020-08-19 12:55:04,557 [main] INFO co.elastic.apm.agent.impl.ElasticApmTracer - Tracer switched to RUNNING state
2020-08-19 12:55:05,545 [elastic-apm-server-healthcheck] INFO co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server is available:
2020-08-19 12:55:05,549 [elastic-apm-server-healthcheck] WARN co.elastic.apm.agent.report.ApmServerHealthChecker - Failed to parse version of APM server https://xxx.apm.us-east-1.aws.cloud.es.io:443/: Unexpected end of JSON input
2020-08-19 12:55:05,550 [elastic-apm-remote-config-poller] ERROR co.elastic.apm.agent.configuration.ApmServerConfigurationSource - Unexpected status 401 while fetching configuration
With a 401 status code, it means there is likely an issue with the api key value, thus just to be sure could you check the agent configuration option value ?
When you generate an API key, you will get a result similar to:
Your server url was reported in the logs (I've since redacted it), and trying to reach it with curl triggered a 502 error, thus I wasn't able to test it myself.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.