# Authentication failed in APM Server in 7.17.4 but works in 8.2.0

**URL:** <https://discuss.elastic.co/t/authentication-failed-in-apm-server-in-7-17-4-but-works-in-8-2-0/307469>\
**Category:** APM\
**Tags:** docker, java\
**Created:** [June 17, 2022, 4:56am UTC](https://discuss.elastic.co/t/authentication-failed-in-apm-server-in-7-17-4-but-works-in-8-2-0/307469 "2022-06-17T04:56:55Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![User28](https://avatars.discourse-cdn.com/v4/letter/u/7cd45c/32.png) [@User28](https://discuss.elastic.co/u/User28)\
**Post date:** [June 17, 2022, 4:56am UTC](https://discuss.elastic.co/t/authentication-failed-in-apm-server-in-7-17-4-but-works-in-8-2-0/307469/1 "2022-06-17T04:56:55Z")

</div>

**Kibana version** : 7.17.4

**Elasticsearch version** : 7.17.4

**APM Server version** : 7.17.4

**APM Agent language and version** : Java APM agent 1.30.0

**Browser version** : Chrome

**Original install method (e.g. download page, yum, deb, from source, etc.) and version**: k8s deployment in azure

**Fresh install or upgraded from other version?** Fresh

**Is there anything special in your setup?** For example, are you using the Logstash or Kafka outputs? Are you using a load balancer in front of the APM Servers? Have you changed index pattern, generated custom templates, changed agent configuration etc. NO

I have tried Elastic APM with ELK stack **8.2.0** and **java apm agent 1.30.0**. It worked well without auth issue. I have disabled SSL and was able to communicate successfully with Elasticsearch from APM Server by just providing username and password. ("elastic"/"elastic").

For some reason, I need to downgrade with version **7.17.4** with the same **java apm agent 1.30.0**. Here I am getting auth issue, its expecting auth headers, but the same works with 8.2.0  
Why i'm not able to communicate with **SUPER USER** (elastic/elastic).

If the auth headers are MUST, then please suggest me a doc for procedures.

Please find below apm server yaml file and logs for reference

**apm server yaml file**

```auto
apiVersion: apm.k8s.elastic.co/v1
kind: ApmServer
metadata:
  name: {{ .Release.Name }}
  namespace: {{ .Release.Namespace }}
spec:
  type: ApmServer
  version: {{ .Values.elasticStackVersion }}
  count: 1
  elasticsearchRef:
    name: {{ .Release.Name }}
  kibanaRef:
    name: {{ .Release.Name }}
  config:
    output:
      elasticsearch:
        username: "elastic"
        password: "elastic"
  logging.level: info
  logging.to_files: true
  logging.files:
    path: /var/log/apm-server
    name: apm-server
    keepfiles: 7
    permissions: 0644

  podTemplate:
    spec:
      containers:
        - name: apm-server
          resources:
            limits:
              memory: 4Gi
              cpu: 1

```

**apm-agent log**

```auto
2022-06-16 08:35:40,230 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - Starting Elastic APM 1.30.0 as apm-translation-service (0.0.3-SNAPSHOT) on Java 17.0.2 Runtime version: 17.0.2+8-86 VM version: 17.0.2+8-86 (Oracle Corporation) Linux 5.4.0-1069-azure
2022-06-16 08:35:40,231 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - service_name: 'apm-translation-service' (source: Java System Properties)
2022-06-16 08:35:40,232 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - server_url: 'https://ip:8200' (source: Java System Properties)
2022-06-16 08:35:40,232 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - verify_server_cert: 'false' (source: Java System Properties)
2022-06-16 08:35:40,232 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - application_packages: 'com.idit.translation' (source: Java System Properties)
2022-06-16 08:35:47,414 [main] INFO co.elastic.apm.agent.impl.ElasticApmTracer - Tracer switched to RUNNING state
2022-06-16 08:35:49,930 [elastic-apm-server-healthcheck] INFO co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server is available: 
2022-06-16 08:35:50,010 [elastic-apm-server-healthcheck] WARN co.elastic.apm.agent.report.ApmServerHealthChecker - Failed to parse version of APM server https://ip:8200/: Unexpected end of JSON input
2022-06-16 08:35:50,017 [elastic-apm-remote-config-poller] ERROR co.elastic.apm.agent.configuration.ApmServerConfigurationSource - Unexpected status 401 while fetching configuration

```

**apm server log**

```auto
{"log.level":"error","@timestamp":"2022-06-17T04:23:01.558Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":60},"message":"authentication failed: missing or improperly formatted Authorization header: expected 'Authorization: Bearer secret_token' or 'Authorization: ApiKey base64(API key ID:API key)'","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-java/1.30.0 (apm-translation-service 0.0.3-SNAPSHOT)","source.address":"ip","http.request.id":"6bb08034-529e-4457-a78c-e76db4a129ed","event.duration":116701,"http.response.status_code":401,"error.message":"authentication failed: missing or improperly formatted Authorization header: expected 'Authorization: Bearer secret_token' or 'Authorization: ApiKey base64(API key ID:API key)'","ecs.version":"1.6.0"}

```

---

<div class="post-metadata">

**Author:** ![Sylvain\_Juge](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylvain_juge/32/55521_2.png) [@Sylvain\_Juge](https://discuss.elastic.co/u/Sylvain_Juge)\
**Post date:** [June 17, 2022, 6:48am UTC](https://discuss.elastic.co/t/authentication-failed-in-apm-server-in-7-17-4-but-works-in-8-2-0/307469/2 "2022-06-17T06:48:54Z")

</div>

Hi @User28 , welcome to the forum !

The agent gets a 401 error when trying to retrieve configuration, this indicates that the credentials are missing/invalid, there are no SSL/TLS errors thus we can assume that communication with the server works as expected.

From your agent configuration, we see that neither `secret_token` nor `api_key` are set, one of them needs to be used for agent authentication , see [Reporter configuration options | APM Java Agent Reference [1.x] | Elastic](https://www.elastic.co/guide/en/apm/agent/java/current/config-reporter.html#config-reporter) for details on how to configure them.

Also, please not that authentication of apm-server on Elasticsearch is distinct from apm agent to apm-server authentication.

I know that it is possible to make apm-server do not ask for any agent authentication through configuration, doing a diff between the two configurations could help to spot any difference here.

Maybe a default value has changed between the two versions (one requiring apm-agent authentication but not the other), hence triggering this behavior.

---

<div class="post-metadata">

**Author:** ![User28](https://avatars.discourse-cdn.com/v4/letter/u/7cd45c/32.png) [@User28](https://discuss.elastic.co/u/User28)\
**Post date:** [June 23, 2022, 6:44am UTC](https://discuss.elastic.co/t/authentication-failed-in-apm-server-in-7-17-4-but-works-in-8-2-0/307469/3 "2022-06-23T06:44:43Z")

</div>

Thanks @Sylvain_Juge  
I created `secret token` from Kibana for the apm integration.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/4/642f01a09cf259136303a10fa2862ac248f2b129.png)  
But still it fails

```auto
2022-06-23 06:34:11,186 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - service_name: 'apm-translation-service' (source: Java System Properties)
2022-06-23 06:34:11,186 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - environment: 'production' (source: Java System Properties)
2022-06-23 06:34:11,186 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - secret_token: 'XXXX' (source: Java System Properties)
2022-06-23 06:34:11,187 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - server_url: 'https://ip:8200' (source: Java System Properties)
2022-06-23 06:34:11,187 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - verify_server_cert: 'false' (source: Java System Properties)
2022-06-23 06:34:11,187 [main] INFO co.elastic.apm.agent.configuration.StartupInfo - application_packages: 'com.idit.translation' (source: Java System Properties)
2022-06-23 06:34:16,580 [main] INFO co.elastic.apm.agent.impl.ElasticApmTracer - Tracer switched to RUNNING state
2022-06-23 06:34:19,179 [elastic-apm-server-healthcheck] INFO co.elastic.apm.agent.report.ApmServerHealthChecker - Elastic APM server is available: 
2022-06-23 06:34:19,180 [elastic-apm-server-healthcheck] WARN co.elastic.apm.agent.report.ApmServerHealthChecker - Failed to parse version of APM server https://10.244.1.72:8200/: Unexpected end of JSON input
2022-06-23 06:34:19,181 [elastic-apm-remote-config-poller] ERROR co.elastic.apm.agent.configuration.ApmServerConfigurationSource - Unexpected status 401 while fetching configuration

```

---

<div class="post-metadata">

**Author:** ![Sylvain\_Juge](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylvain_juge/32/55521_2.png) [@Sylvain\_Juge](https://discuss.elastic.co/u/Sylvain_Juge)\
**Post date:** [June 23, 2022, 7:18am UTC](https://discuss.elastic.co/t/authentication-failed-in-apm-server-in-7-17-4-but-works-in-8-2-0/307469/4 "2022-06-23T07:18:11Z")

</div>

Hi !

I'm not very familiar with Anonymous agent access, what I know is that it is usually only used for RUM agent, which runs in the browser according to documentation: [Anonymous auth configuration options | APM User Guide [master] | Elastic](https://www.elastic.co/guide/en/apm/guide/master/configuration-anonymous.html), if you want to use that with the Java agent you will have to change the defaults.

Have you tried using the API keys and/or secret token instead ?

---

<div class="post-metadata">

**Author:** ![User28](https://avatars.discourse-cdn.com/v4/letter/u/7cd45c/32.png) [@User28](https://discuss.elastic.co/u/User28)\
**Post date:** [June 23, 2022, 7:20am UTC](https://discuss.elastic.co/t/authentication-failed-in-apm-server-in-7-17-4-but-works-in-8-2-0/307469/5 "2022-06-23T07:20:40Z")

</div>

Hey,  
Used secret token only.  
I'm trying with API keys now, but it should work with `secret token` only as per docs

---

<div class="post-metadata">

**Author:** ![Sylvain\_Juge](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/sylvain_juge/32/55521_2.png) [@Sylvain\_Juge](https://discuss.elastic.co/u/Sylvain_Juge)\
**Post date:** [June 23, 2022, 7:49am UTC](https://discuss.elastic.co/t/authentication-failed-in-apm-server-in-7-17-4-but-works-in-8-2-0/307469/6 "2022-06-23T07:49:46Z")

</div>

In this case, from the host that runs the application, could you try one of those `curl` commands to check the validity of the secret token/API key ?

In case of success, you should get the APM server version as reply.

```auto
# When using secret token
curl --request GET \
  --url https://ip:8200 \
  --header 'Authorization: Bearer <secret_token>'
 
# When using API key
curl --request GET \
  --url https://ip:8200 \
  --header 'Authorization: ApiKey <api_key>'

```

---

<div class="post-metadata">

**Author:** ![User28](https://avatars.discourse-cdn.com/v4/letter/u/7cd45c/32.png) [@User28](https://discuss.elastic.co/u/User28)\
**Post date:** [June 27, 2022, 5:07am UTC](https://discuss.elastic.co/t/authentication-failed-in-apm-server-in-7-17-4-but-works-in-8-2-0/307469/7 "2022-06-27T05:07:03Z")

</div>

Thanks Sylvain.  
I am able to authenticate now.

I was deploying apm server with ECK Operator. Operator `generates secret token` and can be fetched from `kubectl get secret/apm-server-quickstart-apm-token -o go-template='{{index .data "secret-token" | base64decode}}`

Docs: [Connect to the APM Server | Elastic Cloud on Kubernetes [master] | Elastic](https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-apm-connecting.html#k8s-apm-secret-token)

Same token can be used for auth.

**Question:** Can `secret token` configurable in `apm-server.yaml` file in kubernetes when deploying with ECK Operator ?

Thanks!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 18, 2022, 1:07am UTC](https://discuss.elastic.co/t/authentication-failed-in-apm-server-in-7-17-4-but-works-in-8-2-0/307469/8 "2022-07-18T01:07:43Z")

</div>

This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.