Elastic APM(8.5.0) - error querying cluster_uuid

Kibana version: 8.5.0

Elasticsearch version: 8.5.0

APM Server version: 8.5.0

APM Agent language and version: Java, 1.35.0

Browser version: Chrome, 105.0.5195.102

Original install method (e.g. download page, yum, deb, from source, etc.) and version: Using ECK Operator 2.5.0

Fresh install or upgraded from other version?
Fresh install
Is there anything special in your setup?

  • This is deployed in kubernetes. No loadbalancers, but exposed node port for apm agents to connect to apm server.

  • No fleet server was configured and no elastic agent deployed

  • Enabled APM integration from the kibana to create ingest pipelines and templates.

Description of the problem including expected versus actual behavior. Please include screenshots (if relevant):
The apm server was deployed using the eck operator and expected to start and ingest traces. But it fails in pre-condition with following error: message":"precondition failed: error querying cluster_uuid: status_code=401

Steps to reproduce:

  1. Install Elasticsearch using eck with following settings enabled
  • xpack.security.enabled: true

  • xpack.security.authc.api_key.enabled: true

  1. Installed Kibana using eck with following settings enabled
  • xpack.encryptedSavedObjects.encryptionKey: "aaaa_bbbb_cccc_dddd_eeee_aaaa_bbbb_cccc"
  1. Installed apm using following definition:
apiVersion: apm.k8s.elastic.co/v1
kind: ApmServer
metadata:
  name: apm-server
  namespace: monitoring
spec:
  version: 8.5.0
  count: 1
  secureSettings:
  - secretName: elasticsearch-es-elastic-user
  http:
    tls:
      selfSignedCertificate:
        disabled: true
  config:
    output:
      elasticsearch:
        hosts: ["elasticsearch-es-http.monitoring:9200"]
        protocol: "https"
        api_key: "5uGxo-V6TEans0vL4G0yqA"
        ssl.certificate_verification: false
        ssl.certificate_authorities: ["/usr/share/apm-server/config/elasticsearch-ca/tls.crt"]
   
    kibana:
      host: kibana-kb-http.monitoring.svc:5601
      protocol: http
      path: /kibana

  podTemplate:
    spec:
      containers:
      - name: apm-server
        volumeMounts:
        - mountPath: /usr/share/apm-server/config/elasticsearch-ca
          name: elasticsearch-ca
          readOnly: true
      volumes:
      - name: elasticsearch-ca
        secret:
          defaultMode: 420
          optional: false
          secretName: elasticsearch-es-http-ca-internal
  • Have also tried with username and password in outpu.elasticsearch but same error

Errors in browser console (if relevant): NA

Provide logs and/or server output (if relevant):
APM Server Logs:
{"log.level":"error","@timestamp":"2023-01-13T06:55:17.053Z","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: error querying cluster_uuid: status_code=401","service.name":"apm-server","ecs.version":"1.6.0"}

Logs from Agent:

{"log.level":"info","@timestamp":"2023-01-13T06:40:34.356Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":61},"message":"request accepted","service.name":"apm-server","url.original":"/intake/v2/events","http.request.method":"POST","user_agent.original":"apm-agent-java/1.35.0 (platform-service)","source.address":"192.168.224.56","http.request.id":"52db6ef7-22af-4275-9fc8-b46372e47ad0","event.duration":10025646449,"http.response.status_code":202,"ecs.version":"1.6.0"}

{"log.level":"error","@timestamp":"2023-01-13T06:40:43.175Z","log.logger":"request","log.origin":{"file.name":"middleware/log_middleware.go","file.line":58},"message":"forbidden request","service.name":"apm-server","url.original":"/config/v1/agents","http.request.method":"POST","user_agent.original":"apm-agent-java/1.35.0 (platform-service)","source.address":"192.168.224.56","http.request.id":"f60243db-c017-4d7c-a385-620f24d90e43","event.duration":276653,"http.response.status_code":403,"error.message":"forbidden request: Agent remote configuration is disabled. Configure the `apm-server.kibana` section in apm-server.yml to enable it. If you are using a RUM agent, you also need to configure the `apm-server.rum` section. If you are not using remote configuration, you can safely ignore this error.","ecs.version":"1.6.0"}

Is there a (good) reason for the custom TLS and API key configuration? This looks a lot more complicated than it would need to be and I wonder why.

Also the 401 error points in that direction and the "Important: Updating to 7.0 or higher" also sounds to me like the APM server cannot properly connect to Elasticsearch.

Hi @ragur ,

A 401 indicates that Elasticsearch was not able to find the corresponding api key. Looking at the definition you provided, the output.elasticsearch.api_key appears to be in a wrong format. It should be id:api_key, where id and api_key corresponds to the fields in Create API key API json output.

For example, if Elasticsearch Create API key API returns:

{"id":"j7EzuoUBK-FkvYp7ebv6","name":"apm-user","api_key":"Gf20hhEeQjOHSZZKzcH3ow","encoded":"ajdFenVvVUJLLUZrdllwN2VidjY6R2YyMGhoRWVRak9IU1paS3pjSDNvdw=="}

In k8s definition / apm server yml config, output.elasticsearch.api_key should be j7EzuoUBK-FkvYp7ebv6:Gf20hhEeQjOHSZZKzcH3ow.

Please retry with the right API key in the definition and see if it still returns an error.


Have also tried with username and password in outpu.elasticsearch but same error

If fixing the API key does not work, could you please show the server logs when using username and password in output.elasticsearch? It is rather unlikely for a username and password output.elasticsearch to return a 401.

Hi,
Thanks for the reply. I did try with api-key option and unfortunately it didn't work. I created a api-key for the user elastic and the id and api key was in following format:

{
"id": xxxxxxxx
"api_key": xxxxxxxxxxxxxxxxx
}

With the above api key it returned the same error.

When i retrieve the api key using GET _security/api_key, the api keys are generated for username: elastic/kibana. Not sure why the username is set as elastic/kibana.

It was returning 401 with username and password that's when i tried with api key option which resulted in the same error.

Do you mind posting just the privilege part of the output of GET _security/api_key to confirm that there are sufficient cluster and indices privileges? Please do not share any sensitive information on this forum.

The api key / user should have the following privileges in order for apm server to run:

  {
    "cluster": [
      "monitor",
      "manage_own_api_key"
    ],
    "indices": [
      {
        "names": [
          "logs-*",
          "metrics-*",
          "traces-*",
          ".logs-endpoint.diagnostic.collection-*",
          ".logs-endpoint.action.responses-*"
        ],
        "privileges": [
          "write",
          "create_index",
          "auto_configure"
        ],
        "allow_restricted_indices": false
      }
    ]
  }

Hi @ragur , I appreciate the examples for the definition, id and api keys but for security reasons please invalidate the API keys shared in the post and do not post any sensitive information on the forum.

Hi,
These were sample key and id. Anyways i have updated the thread. The resources field was empty for the created api key and there was no documentation or mention about these privileges to be set for cluster and indices.

I have fixed the issue by moving to a fleet manged elastic agents rather than deploying an kind: ApmServer.

I have one more follow up question, not sure should i create a new ticket or discuss here. We have custom path for Kibana and there is no way to set the kibana custom path in elastic agent definition like we do it in the ApmServer. Or there might be a configuration, which i am not aware of.

Hi,

Glad to know that the issue has been resolved. We are going to update the docs about the required cluster and indices privileges.

For the question around Kibana custom path, please create a new topic under Elastic Agent or Kibana and you will get better help over there.

1 Like