Elastic APM (8.1) with Docker compose: error querying cluster_uuid: status_code=401

Elastic Observability
1

Hello,

I'm trying to run Elasticsearch, Kibana and Elastic APM with Docker Compose. The final goal is to send telemetry data to the Elastic Stack through the OpenTelemetry collector.

After adding the APM Integration through Kibana, I see this error on the Elastic APM output logs:

{"log.level":"error","@timestamp":"2022-03-12T16:57:04.570Z","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: error querying cluster_uuid: status_code=401","service.name":"apm-server","ecs.version":"1.6.0"}

The error appears continually every 5 seconds and I don't understand if it is blocking.

This my docker-compose:

elasticsearch:
    container_name: es-container
    image: docker.elastic.co/elasticsearch/elasticsearch:8.1.0
    environment:
      - "discovery.type=single-node"
      - xpack.security.enabled=true
      - xpack.security.authc.api_key.enabled=true
      - ELASTIC_PASSWORD=elastic_test
    volumes:
      - elasticsearch_data:/usr/share/elasticsearch/data
    ports:
      - 9200:9200

  kibana:
    container_name: kb-container
    image: docker.elastic.co/kibana/kibana:8.1.0
    environment:
      - ELASTICSEARCH_HOSTS=http://elasticsearch:9200
      - ELASTICSEARCH_USERNAME=kibana_system
      - ELASTICSEARCH_PASSWORD=kibana_system
      - XPACK_SECURITY_ENABLED=true
      - XPACK_ENCRYPTEDSAVEDOBJECTS_ENCRYPTIONKEY=aaaa_bbbb_cccc_dddd_eeee_aaaa_bbbb_cccc
    volumes:
      - kibana_data:/usr/share/kibana/data
    ports:
      - 5601:5601
    depends_on:
      - elasticsearch

  apm-es-server:
    container_name: apm-es-container
    image: docker.elastic.co/apm/apm-server:8.1.0
    environment:
      - output.elasticsearch.hosts=["elasticsearch:9200"]
    ports:
      - "8200:8200"
    depends_on:
      - elasticsearch

Any advice?

2

I am having the same issue. Any luck?
Edit: I was able to resolve this issue by specifying the correct protocol and port number. Elasticsearch Output was originally set to http, switching it to https resolved the issue for me. But now I am getting a different error.
{"log.level":"error","@timestamp":"2022-04-06T16:34:55.701Z","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":62},"message":"precondition 'apm integration installed' failed: error querying Elasticsearch for integration index templates: unexpected HTTP status: 404 Not Found ({"error":{"root_cause":[{"type":"resource_not_found_exception","reason":"index template matching [traces-apm.sampled] not found"}],"type":"resource_not_found_exception","reason":"index template matching [traces-apm.sampled] not found"},"status":404}): to remediate, please install the apm integration: https://ela.st/apm-integration-quickstart","service.name":"apm-server","ecs.version":"1.6.0"}

How Can I use the legacy APM setup on version 8.0.0 ?

3

Has anyone figured this one out? I get the:

precondition failed: error querying cluster_uuid: status_code=401

message as well. I did confirm connectivity between the APM server and Elasticsearch:

apm-server@APM:~$ curl --cacert certs/http_ca.crt -u elastic https://elasticsearch:9200
Enter host password for user 'elastic':
{
"name" : "Elasticsearch",
"cluster_name" : "docker-cluster",
"cluster_uuid" : "gEHs0IMOR8GOprOt_kj7eQ",
"version" : {
"number" : "8.1.2",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "31df9689e80bad366ac20176aa7f2371ea5eb4c1",
"build_date" : "2022-03-29T21:18:59.991429448Z",
"build_snapshot" : false,
"lucene_version" : "9.0.0",
"minimum_wire_compatibility_version" : "7.17.0",
"minimum_index_compatibility_version" : "7.0.0"
},
"tagline" : "You Know, for Search"
}
apm-server@APM:~$

my apm-server.yml contains:

apm-server:
host: "localhost:8200"
api_key:
enabled: false
limit: 1000
rum:
enabled: true

output:
Elasticsearch:
hosts: ["Elasticsearch:9200"]
protocol: "https"
usernname: "elastic"
password: "changeme"
ssl:
certificate_authorities: certs/http_ca.crt

queue.mem.events: 4096

max_procs: 4

4

For my setup (APM as a docker container) this apm-server.yml resolved my issue:

apm-server:
host: "0.0.0.0:8200"
api_key:
enabled: false
limit: 1000
rum:
enabled: true

output:
Elasticsearch:
hosts: ["Elasticsearch:9200"]
protocol: "https"
api_key: LUy-LoAB5bz5ukdMOT0c:ptJZH521TlutsTuHbRvdhQ
ssl.certificate_authorities: certs/http_ca.crt
ssl.certificate_verification: false
queue.mem.events: 4096

max_procs: 4

for me one of the keys was getting the API key correct, that is with an ID:KEY pair. I used this curl command to generate the key:

curl -X POST -H 'Content-Type: application/json' --cacert http_ca.crt -u elastic -d '{"name": "my-api-key","role_descriptors": {} }' https://localhost:9200

I pulled the cacert from the Elasticsearch docker container with this command:

docker cp Elasticsearch:/usr/share/Elasticsearch/config/certs/http_ca.crt .

which I pushed to the APM docker image with a -v command:

-v /usr/local/ELK/manual/APM/config/certs/:/usr/share/apm-server/certs/

in my docker run command