# Elastic APM(8.5.0) - error querying cluster\_uuid
**URL:** https://discuss.elastic.co/t/elastic-apm-8-5-0-error-querying-cluster-uuid/323097
**Category:** APM
**Tags:** server
**Created:** [January 13, 2023, 7:33am UTC](https://discuss.elastic.co/t/elastic-apm-8-5-0-error-querying-cluster-uuid/323097 "2023-01-13T07:33:09Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@ragur](https://discuss.elastic.co/u/ragur)
#### Post date: [January 13, 2023, 7:33am UTC](https://discuss.elastic.co/t/elastic-apm-8-5-0-error-querying-cluster-uuid/323097/1 "2023-01-13T07:33:09Z")
**Kibana version** : 8.5.0
**Elasticsearch version** : 8.5.0
**APM Server version** : 8.5.0
**APM Agent language and version** : Java, 1.35.0
**Browser version** : Chrome, 105.0.5195.102
**Original install method (e.g. download page, yum, deb, from source, etc.) and version**: Using ECK Operator 2.5.0
**Fresh install or upgraded from other version?**
Fresh install
**Is there anything special in your setup?**
- This is deployed in kubernetes. No loadbalancers, but exposed node port for apm agents to connect to apm server.
- No fleet server was configured and no elastic agent deployed
- Enabled APM integration from the kibana to create ingest pipelines and templates.
**Description of the problem including expected versus actual behavior. Please include screenshots (if relevant)**:
The apm server was deployed using the eck operator and expected to start and ingest traces. But it fails in pre-condition with following error: message":"precondition failed: error querying cluster\_uuid: status\_code=401
**Steps to reproduce** :
1. Install Elasticsearch using eck with following settings enabled
- xpack.security.enabled: true
- xpack.security.authc.api\_key.enabled: true
1. Installed Kibana using eck with following settings enabled
- xpack.encryptedSavedObjects.encryptionKey: "aaaa\_bbbb\_cccc\_dddd\_eeee\_aaaa\_bbbb\_cccc"
1. Installed apm using following definition:
```auto
apiVersion: apm.k8s.elastic.co/v1
kind: ApmServer
metadata:
name: apm-server
namespace: monitoring
spec:
version: 8.5.0
count: 1
secureSettings:
- secretName: elasticsearch-es-elastic-user
http:
tls:
selfSignedCertificate:
disabled: true
config:
output:
elasticsearch:
hosts: ["elasticsearch-es-http.monitoring:9200"]
protocol: "https"
api_key: "5uGxo-V6TEans0vL4G0yqA"
ssl.certificate_verification: false
ssl.certificate_authorities: ["/usr/share/apm-server/config/elasticsearch-ca/tls.crt"]
kibana:
host: kibana-kb-http.monitoring.svc:5601
protocol: http
path: /kibana
podTemplate:
spec:
containers:
- name: apm-server
volumeMounts:
- mountPath: /usr/share/apm-server/config/elasticsearch-ca
name: elasticsearch-ca
readOnly: true
volumes:
- name: elasticsearch-ca
secret:
defaultMode: 420
optional: false
secretName: elasticsearch-es-http-ca-internal
```
- Have also tried with username and password in outpu.elasticsearch but same error
**Errors in browser console (if relevant)**: NA
**Provide logs and/or server output (if relevant)**:
**APM Server Logs:**
`{"log.level":"error","@timestamp":"2023-01-13T06:55:17.053Z","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: error querying cluster_uuid: status_code=401","service.name":"apm-server","ecs.version":"1.6.0"}`
**Logs from Agent:**
```auto
{"log.level":"info","@timestamp":"2023-01-13T06:40:34.356Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-java/1.35.0 (platform-service)","source.address":"192.168.224.56","http.request.id":"52db6ef7-22af-4275-9fc8-b46372e47ad0","event.duration":10025646449,"http.response.status_code":202,"ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-01-13T06:40:43.175Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":58},"message":"forbidden request","service.name":"apm-server","url.original":"/config/v1/agents","http.request.method":"POST","user_agent.original":"apm-agent-java/1.35.0 (platform-service)","source.address":"192.168.224.56","http.request.id":"f60243db-c017-4d7c-a385-620f24d90e43","event.duration":276653,"http.response.status_code":403,"error.message":"forbidden request: Agent remote configuration is disabled. Configure the `apm-server.kibana` section in apm-server.yml to enable it. If you are using a RUM agent, you also need to configure the `apm-server.rum` section. If you are not using remote configuration, you can safely ignore this error.","ecs.version":"1.6.0"}
```
---
### Author:  [@xeraa](https://discuss.elastic.co/u/xeraa)
#### Post date: [January 16, 2023, 12:03am UTC](https://discuss.elastic.co/t/elastic-apm-8-5-0-error-querying-cluster-uuid/323097/2 "2023-01-16T00:03:34Z")
Is there a (good) reason for the custom TLS and API key configuration? This looks a lot more complicated than it would need to be and I wonder why.
Also the 401 error points in that direction and the "Important: Updating to 7.0 or higher" also sounds to me like the APM server cannot properly connect to Elasticsearch.
---
### Author:  [@Carson\_Ip](https://discuss.elastic.co/u/Carson_Ip)
#### Post date: [January 16, 2023, 11:17am UTC](https://discuss.elastic.co/t/elastic-apm-8-5-0-error-querying-cluster-uuid/323097/3 "2023-01-16T11:17:48Z")
Hi @ragur ,
A 401 indicates that Elasticsearch was not able to find the corresponding api key. Looking at the definition you provided, the `output.elasticsearch.api_key` appears to be in a wrong format. It should be `id:api_key`, where `id` and `api_key` corresponds to the fields in Create API key API json output.
For example, if Elasticsearch Create API key API returns:
```auto
{"id":"j7EzuoUBK-FkvYp7ebv6","name":"apm-user","api_key":"Gf20hhEeQjOHSZZKzcH3ow","encoded":"ajdFenVvVUJLLUZrdllwN2VidjY6R2YyMGhoRWVRak9IU1paS3pjSDNvdw=="}
```
In k8s definition / apm server yml config, `output.elasticsearch.api_key` should be `j7EzuoUBK-FkvYp7ebv6:Gf20hhEeQjOHSZZKzcH3ow`.
Please retry with the right API key in the definition and see if it still returns an error.
* * *
> Have also tried with username and password in outpu.elasticsearch but same error
If fixing the API key does not work, could you please show the server logs when using username and password in `output.elasticsearch`? It is rather unlikely for a username and password `output.elasticsearch` to return a 401.
---
### Author:  [@ragur](https://discuss.elastic.co/u/ragur)
#### Post date: [January 16, 2023, 12:25pm UTC](https://discuss.elastic.co/t/elastic-apm-8-5-0-error-querying-cluster-uuid/323097/4 "2023-01-16T12:25:58Z")
Hi,
Thanks for the reply. I did try with api-key option and unfortunately it didn't work. I created a api-key for the user **elastic** and the id and api key was in following format:
```auto
{
"id": xxxxxxxx
"api_key": xxxxxxxxxxxxxxxxx
}
```
With the above api key it returned the same error.
When i retrieve the api key using GET \_security/api\_key, the api keys are generated for username: elastic/kibana. Not sure why the username is set as elastic/kibana.
It was returning 401 with username and password that's when i tried with api key option which resulted in the same error.
---
### Author:  [@Carson\_Ip](https://discuss.elastic.co/u/Carson_Ip)
#### Post date: [January 16, 2023, 4:00pm UTC](https://discuss.elastic.co/t/elastic-apm-8-5-0-error-querying-cluster-uuid/323097/5 "2023-01-16T16:00:31Z")
Do you mind posting just the privilege part of the output of GET \_security/api\_key to confirm that there are sufficient cluster and indices privileges? Please do not share any sensitive information on this forum.
The api key / user should have the following privileges in order for apm server to run:
```auto
{
"cluster": [
"monitor",
"manage_own_api_key"
],
"indices": [
{
"names": [
"logs-*",
"metrics-*",
"traces-*",
".logs-endpoint.diagnostic.collection-*",
".logs-endpoint.action.responses-*"
],
"privileges": [
"write",
"create_index",
"auto_configure"
],
"allow_restricted_indices": false
}
]
}
```
---
### Author:  [@Carson\_Ip](https://discuss.elastic.co/u/Carson_Ip)
#### Post date: [January 16, 2023, 7:52pm UTC](https://discuss.elastic.co/t/elastic-apm-8-5-0-error-querying-cluster-uuid/323097/6 "2023-01-16T19:52:51Z")
Hi @ragur , I appreciate the examples for the definition, id and api keys but for security reasons please invalidate the API keys shared in the post and do not post any sensitive information on the forum.
---
### Author:  [@ragur](https://discuss.elastic.co/u/ragur)
#### Post date: [January 17, 2023, 2:15am UTC](https://discuss.elastic.co/t/elastic-apm-8-5-0-error-querying-cluster-uuid/323097/7 "2023-01-17T02:15:57Z")
Hi,
These were sample key and id. Anyways i have updated the thread. The resources field was empty for the created api key and there was no documentation or mention about these privileges to be set for cluster and indices.
I have fixed the issue by moving to a fleet manged elastic agents rather than deploying an kind: ApmServer.
I have one more follow up question, not sure should i create a new ticket or discuss here. We have custom path for Kibana and there is no way to set the kibana custom path in elastic agent definition like we do it in the ApmServer. Or there might be a configuration, which i am not aware of.
---
### Author:  [@Carson\_Ip](https://discuss.elastic.co/u/Carson_Ip)
#### Post date: [January 17, 2023, 9:32am UTC](https://discuss.elastic.co/t/elastic-apm-8-5-0-error-querying-cluster-uuid/323097/8 "2023-01-17T09:32:41Z")
Hi,
Glad to know that the issue has been resolved. We are going to update the docs about the required cluster and indices privileges.
For the question around Kibana custom path, please create a new topic under Elastic Agent or Kibana and you will get better help over there.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 7, 2023, 5:33am UTC](https://discuss.elastic.co/t/elastic-apm-8-5-0-error-querying-cluster-uuid/323097/9 "2023-02-07T05:33:29Z")
This topic was automatically closed 20 days after the last reply. New replies are no longer allowed.