Authentication failed in APM Server in 7.17.4 but works in 8.2.0

User28 · June 17, 2022, 4:56am

Kibana version: 7.17.4

Elasticsearch version: 7.17.4

APM Server version: 7.17.4

APM Agent language and version: Java APM agent 1.30.0

Browser version: Chrome

Original install method (e.g. download page, yum, deb, from source, etc.) and version: k8s deployment in azure

Fresh install or upgraded from other version? Fresh

Is there anything special in your setup? For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc. NO

I have tried Elastic APM with ELK stack 8.2.0 and java apm agent 1.30.0. It worked well without auth issue. I have disabled SSL and was able to communicate successfully with Elasticsearch from APM Server by just providing username and password. ("elastic"/"elastic").

For some reason, I need to downgrade with version 7.17.4 with the same java apm agent 1.30.0. Here I am getting auth issue, its expecting auth headers, but the same works with 8.2.0
Why i'm not able to communicate with SUPER USER (elastic/elastic).

If the auth headers are MUST, then please suggest me a doc for procedures.

Please find below apm server yaml file and logs for reference

apm server yaml file

apiVersion: apm.k8s.elastic.co/v1
kind: ApmServer
metadata:
  name: {{ .Release.Name }}
  namespace: {{ .Release.Namespace }}
spec:
  type: ApmServer
  version: {{ .Values.elasticStackVersion }}
  count: 1
  elasticsearchRef:
    name: {{ .Release.Name }}
  kibanaRef:
    name: {{ .Release.Name }}
  config:
    output:
      elasticsearch:
        username: "elastic"
        password: "elastic"
  logging.level: info
  logging.to_files: true
  logging.files:
    path: /var/log/apm-server
    name: apm-server
    keepfiles: 7
    permissions: 0644

  podTemplate:
    spec:
      containers:
        - name: apm-server
          resources:
            limits:
              memory: 4Gi
              cpu: 1

apm-agent log

2022-06-16 08:35:40,230 [main] INFO  co.elastic.apm.agent.configuration.StartupInfo - Starting Elastic APM 1.30.0 as apm-translation-service (0.0.3-SNAPSHOT) on Java 17.0.2 Runtime version: 17.0.2+8-86 VM version: 17.0.2+8-86 (Oracle Corporation) Linux 5.4.0-1069-azure
2022-06-16 08:35:40,231 [main] INFO  co.elastic.apm.agent.configuration.StartupInfo - service_name: 'apm-translation-service' (source: Java System Properties)
2022-06-16 08:35:40,232 [main] INFO  co.elastic.apm.agent.configuration.StartupInfo - server_url: 'https://ip:8200' (source: Java System Properties)
2022-06-16 08:35:40,232 [main] INFO  co.elastic.apm.agent.configuration.StartupInfo - verify_server_cert: 'false' (source: Java System Properties)
2022-06-16 08:35:40,232 [main] INFO  co.elastic.apm.agent.configuration.StartupInfo - application_packages: 'com.idit.translation' (source: Java System Properties)
2022-06-16 08:35:47,414 [main] INFO  co.elastic.apm.agent.impl.ElasticApmTracer - Tracer switched to RUNNING state
2022-06-16 08:35:49,930 [elastic-apm-server-healthcheck] INFO  co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server is available: 
2022-06-16 08:35:50,010 [elastic-apm-server-healthcheck] WARN  co.elastic.apm.agent.report.ApmServerHealthChecker - Failed to parse version of APM server https://ip:8200/: Unexpected end of JSON input
2022-06-16 08:35:50,017 [elastic-apm-remote-config-poller] ERROR co.elastic.apm.agent.configuration.ApmServerConfigurationSource - Unexpected status 401 while fetching configuration

apm server log

{"log.level":"error","@timestamp":"2022-06-17T04:23:01.558Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":60},"message":"authentication failed: missing or improperly formatted Authorization header: expected 'Authorization: Bearer secret_token' or 'Authorization: ApiKey base64(API key ID:API key)'","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-java/1.30.0 (apm-translation-service 0.0.3-SNAPSHOT)","source.address":"ip","http.request.id":"6bb08034-529e-4457-a78c-e76db4a129ed","event.duration":116701,"http.response.status_code":401,"error.message":"authentication failed: missing or improperly formatted Authorization header: expected 'Authorization: Bearer secret_token' or 'Authorization: ApiKey base64(API key ID:API key)'","ecs.version":"1.6.0"}

Sylvain_Juge · June 17, 2022, 6:48am

Hi @User28 , welcome to the forum !

The agent gets a 401 error when trying to retrieve configuration, this indicates that the credentials are missing/invalid, there are no SSL/TLS errors thus we can assume that communication with the server works as expected.

From your agent configuration, we see that neither secret_token nor api_key are set, one of them needs to be used for agent authentication , see Reporter configuration options | APM Java Agent Reference [1.x] | Elastic for details on how to configure them.

Also, please not that authentication of apm-server on Elasticsearch is distinct from apm agent to apm-server authentication.

I know that it is possible to make apm-server do not ask for any agent authentication through configuration, doing a diff between the two configurations could help to spot any difference here.

Maybe a default value has changed between the two versions (one requiring apm-agent authentication but not the other), hence triggering this behavior.

User28 · June 23, 2022, 6:44am

Thanks @Sylvain_Juge
I created secret token from Kibana for the apm integration.

But still it fails

2022-06-23 06:34:11,186 [main] INFO  co.elastic.apm.agent.configuration.StartupInfo - service_name: 'apm-translation-service' (source: Java System Properties)
2022-06-23 06:34:11,186 [main] INFO  co.elastic.apm.agent.configuration.StartupInfo - environment: 'production' (source: Java System Properties)
2022-06-23 06:34:11,186 [main] INFO  co.elastic.apm.agent.configuration.StartupInfo - secret_token: 'XXXX' (source: Java System Properties)
2022-06-23 06:34:11,187 [main] INFO  co.elastic.apm.agent.configuration.StartupInfo - server_url: 'https://ip:8200' (source: Java System Properties)
2022-06-23 06:34:11,187 [main] INFO  co.elastic.apm.agent.configuration.StartupInfo - verify_server_cert: 'false' (source: Java System Properties)
2022-06-23 06:34:11,187 [main] INFO  co.elastic.apm.agent.configuration.StartupInfo - application_packages: 'com.idit.translation' (source: Java System Properties)
2022-06-23 06:34:16,580 [main] INFO  co.elastic.apm.agent.impl.ElasticApmTracer - Tracer switched to RUNNING state
2022-06-23 06:34:19,179 [elastic-apm-server-healthcheck] INFO  co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server is available: 
2022-06-23 06:34:19,180 [elastic-apm-server-healthcheck] WARN  co.elastic.apm.agent.report.ApmServerHealthChecker - Failed to parse version of APM server https://10.244.1.72:8200/: Unexpected end of JSON input
2022-06-23 06:34:19,181 [elastic-apm-remote-config-poller] ERROR co.elastic.apm.agent.configuration.ApmServerConfigurationSource - Unexpected status 401 while fetching configuration

Sylvain_Juge · June 23, 2022, 7:18am

Hi !

I'm not very familiar with Anonymous agent access, what I know is that it is usually only used for RUM agent, which runs in the browser according to documentation: Anonymous auth configuration options | APM User Guide [master] | Elastic, if you want to use that with the Java agent you will have to change the defaults.

Have you tried using the API keys and/or secret token instead ?

User28 · June 23, 2022, 7:20am

Hey,
Used secret token only.
I'm trying with API keys now, but it should work with secret token only as per docs

Sylvain_Juge · June 23, 2022, 7:49am

In this case, from the host that runs the application, could you try one of those curl commands to check the validity of the secret token/API key ?

In case of success, you should get the APM server version as reply.

# When using secret token
curl --request GET \
  --url https://ip:8200 \
  --header 'Authorization: Bearer <secret_token>'
 
# When using API key
curl --request GET \
  --url https://ip:8200 \
  --header 'Authorization: ApiKey <api_key>'

User28 · June 27, 2022, 5:07am

Thanks Sylvain.
I am able to authenticate now.

I was deploying apm server with ECK Operator. Operator generates secret token and can be fetched from kubectl get secret/apm-server-quickstart-apm-token -o go-template='{{index .data "secret-token" | base64decode}}

Docs: Connect to the APM Server | Elastic Cloud on Kubernetes [master] | Elastic

Same token can be used for auth.

Question: Can secret token configurable in apm-server.yaml file in kubernetes when deploying with ECK Operator ?

Thanks!

system · July 18, 2022, 1:07am

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
APM Server failed to index event (security_exception): action [indices:admin/auto_create] is unauthorized for user APM elastic-stack-security , server	1	786	April 22, 2022
APM-Server unable to connect to elasticsearch APM server	3	1392	August 22, 2021
APM server can't access elasticsearch APM nodejs , server	1	365	May 14, 2021
Apm-server failed connect to ElasticSearch with Xpack enabled (401 Unauthorized ) APM	11	3964	August 6, 2019
[apm-reporter] ERROR co.elastic.apm.agent.report.IntakeV2ReportingEventHandler - Error sending data to APM server: cannot retry due to server authentication, in streaming mode, response code is 401 APM java	7	4043	February 24, 2020

Authentication failed in APM Server in 7.17.4 but works in 8.2.0

Related topics