Standalone apm-server with elasticsearch 8 in docker not able to connect

[AlanMark (Alan Mark Kristensen)]

Elasticsearch version: 8.5.0

APM Server version: 8.5.2

Original install method (e.g. download page, yum, deb, from source, etc.) and version: docker

Fresh install or upgraded from other version?: fresh

Setup:
Attempting to run apm-server standalone, ie. not using an elastic agent, and on docker. This worked fine in 7.x, but in 8.x I'm having all sorts of problems.

Using the following setup:

docker run -d \
  -p 8200:8200 \
  --network=elastic \
  --name=apm-server \
  --user=apm-server \
  --volume="$(pwd)/ca.crt:/usr/share/apm-server/ca.crt:ro" \
  docker.elastic.co/apm/apm-server:8.5.2 \
  --strict.perms=false -e \
  -E output.elasticsearch.hosts=["https://es00:9200"] \
  -E output.elasticsearch.api_key="some_api_key_for_elastic_user" \
  -E output.elasticsearch.ssl.certificate_authorities="/usr/share/apm-server/ca.crt"

Elasticsearch is running on the same cluster in the same network. I am using the same CA. The API key is generated with the elastic master user.

Problem:
APM is unable to connect, and it gives a very non-descript error that looks faulty (notice the missing uuid):

apm-server    | {"log.level":"error","@timestamp":"2022-11-27T20:26:57.297Z","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: error querying cluster_uuid: status_code=401","service.name":"apm-server","ecs.version":"1.6.0"}

I have verified that I can reach both containers from eachothers hostnames on the docker network.

I also get the exact same error if i use username / password approach with the elastic user.

I'm aware that the elastic user is bad practice here - I'm just working in a test environment and need to verify settings before hardening.

The remaining log looks like this:

apm-server    | {"log.level":"info","@timestamp":"2022-11-27T20:26:32.287Z","log.origin":{"file.name":"instance/beat.go","file.line":708},"message":"Home path: [/usr/share/apm-server] Config path: [/usr/share/apm-server] Data path: [/usr/share/apm-server/data] Logs path: [/usr/share/apm-server/logs]","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server    | {"log.level":"debug","@timestamp":"2022-11-27T20:26:32.287Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":766},"message":"Beat metadata path: /usr/share/apm-server/data/meta.json","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server    | {"log.level":"info","@timestamp":"2022-11-27T20:26:32.287Z","log.origin":{"file.name":"instance/beat.go","file.line":716},"message":"Beat ID: be37df93-d182-4a1a-822a-777e4f97548b","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server    | {"log.level":"info","@timestamp":"2022-11-27T20:26:32.288Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1082},"message":"Beat info","service.name":"apm-server","system_info":{"beat":{"path":{"config":"/usr/share/apm-server","data":"/usr/share/apm-server/data","home":"/usr/share/apm-server","logs":"/usr/share/apm-server/logs"},"type":"apm-server","uuid":"be37df93-d182-4a1a-822a-777e4f97548b"},"ecs.version":"1.6.0"}}
apm-server    | {"log.level":"info","@timestamp":"2022-11-27T20:26:32.288Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1091},"message":"Build info","service.name":"apm-server","system_info":{"build":{"commit":"603b5878ad5a02225c03970741b87fb7f8c38e4a","libbeat":"8.5.2","time":"2022-11-17T10:57:37.000+0100","version":"8.5.2"},"ecs.version":"1.6.0"}}
apm-server    | {"log.level":"info","@timestamp":"2022-11-27T20:26:32.288Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1094},"message":"Go runtime info","service.name":"apm-server","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":32,"version":"go1.18.5"},"ecs.version":"1.6.0"}}
apm-server    | {"log.level":"info","@timestamp":"2022-11-27T20:26:32.288Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1098},"message":"Host info","service.name":"apm-server","system_info":{SCRUBBED}
apm-server    | {"log.level":"info","@timestamp":"2022-11-27T20:26:32.288Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":1127},"message":"Process info","service.name":"apm-server","system_info":{"process":{"capabilities":{"inheritable":null,"permitted":null,"effective":null,"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null},"cwd":"/usr/share/apm-server","exe":"/usr/share/apm-server/apm-server","name":"apm-server","pid":7,"ppid":1,"seccomp":{"mode":"filter","no_new_privs":false},"start_time":"2022-11-27T20:26:31.910Z"},"ecs.version":"1.6.0"}}
apm-server    | {"log.level":"info","@timestamp":"2022-11-27T20:26:32.288Z","log.origin":{"file.name":"instance/beat.go","file.line":294},"message":"Setup Beat: apm-server; Version: 8.5.2","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server    | {"log.level":"debug","@timestamp":"2022-11-27T20:26:32.289Z","log.logger":"beat","log.origin":{"file.name":"instance/beat.go","file.line":319},"message":"Initializing output plugins","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server    | {"log.level":"warn","@timestamp":"2022-11-27T20:26:32.290Z","log.logger":"cfgwarn","log.origin":{"file.name":"tlscommon/config.go","file.line":102},"message":"DEPRECATED: Treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present is going to be removed. Please update your certificates if needed. Will be removed in version: 8.0.0","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server    | {"log.level":"debug","@timestamp":"2022-11-27T20:26:32.290Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls.go","file.line":172},"message":"Successfully loaded CA certificate: /usr/share/apm-server/ca.crt","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server    | {"log.level":"info","@timestamp":"2022-11-27T20:26:32.290Z","log.logger":"esclientleg","log.origin":{"file.name":"eslegclient/connection.go","file.line":108},"message":"elasticsearch url: https://es00:9200","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server    | {"log.level":"debug","@timestamp":"2022-11-27T20:26:32.291Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls.go","file.line":172},"message":"Successfully loaded CA certificate: /usr/share/apm-server/ca.crt","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server    | {"log.level":"debug","@timestamp":"2022-11-27T20:26:32.291Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/consumer.go","file.line":98},"message":"start pipeline event consumer","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server    | {"log.level":"info","@timestamp":"2022-11-27T20:26:32.291Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/module.go","file.line":113},"message":"Beat name: 825bd2425dff","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server    | {"log.level":"debug","@timestamp":"2022-11-27T20:26:32.291Z","log.logger":"publisher","log.origin":{"file.name":"pipeline/queue_reader.go","file.line":49},"message":"pipeline event consumer queue reader: start","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server    | {"log.level":"info","@timestamp":"2022-11-27T20:26:32.291Z","log.origin":{"file.name":"instance/beat.go","file.line":471},"message":"apm-server start running.","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server    | {"log.level":"info","@timestamp":"2022-11-27T20:26:32.292Z","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":142},"message":"Listening on: [::]:8200","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server    | {"log.level":"info","@timestamp":"2022-11-27T20:26:32.292Z","log.logger":"beater","log.origin":{"file.name":"beater/beater.go","file.line":1162},"message":"maxprocs: Leaving GOMAXPROCS=32: CPU quota undefined","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server    | {"log.level":"debug","@timestamp":"2022-11-27T20:26:32.292Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls.go","file.line":172},"message":"Successfully loaded CA certificate: /usr/share/apm-server/ca.crt","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server    | {"log.level":"debug","@timestamp":"2022-11-27T20:26:32.292Z","log.logger":"tls","log.origin":{"file.name":"tlscommon/tls.go","file.line":172},"message":"Successfully loaded CA certificate: /usr/share/apm-server/ca.crt","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server    | {"log.level":"info","@timestamp":"2022-11-27T20:26:32.292Z","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":40},"message":"blocking ingestion until all preconditions are satisfied","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server    | {"log.level":"info","@timestamp":"2022-11-27T20:26:32.293Z","log.logger":"beater","log.origin":{"file.name":"apm-server/main.go","file.line":71},"message":"creating transaction metrics aggregation with config: {Interval:1m0s MaxTransactionGroups:10000 HDRHistogramSignificantFigures:2}","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server    | {"log.level":"info","@timestamp":"2022-11-27T20:26:32.296Z","log.logger":"beater","log.origin":{"file.name":"apm-server/main.go","file.line":86},"message":"creating service destinations aggregation with config: {Interval:1m0s MaxGroups:10000}","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server    | {"log.level":"info","@timestamp":"2022-11-27T20:26:32.296Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":133},"message":"Path / added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server    | {"log.level":"info","@timestamp":"2022-11-27T20:26:32.296Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":133},"message":"Path /config/v1/agents added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server    | {"log.level":"info","@timestamp":"2022-11-27T20:26:32.296Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":133},"message":"Path /config/v1/rum/agents added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server    | {"log.level":"info","@timestamp":"2022-11-27T20:26:32.296Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":133},"message":"Path /intake/v2/rum/events added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server    | {"log.level":"info","@timestamp":"2022-11-27T20:26:32.296Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":133},"message":"Path /intake/v3/rum/events added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server    | {"log.level":"info","@timestamp":"2022-11-27T20:26:32.296Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":133},"message":"Path /intake/v2/events added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server    | {"log.level":"info","@timestamp":"2022-11-27T20:26:32.296Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":133},"message":"Path /v1/traces added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server    | {"log.level":"info","@timestamp":"2022-11-27T20:26:32.296Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":133},"message":"Path /v1/metrics added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server    | {"log.level":"info","@timestamp":"2022-11-27T20:26:32.296Z","log.logger":"handler","log.origin":{"file.name":"api/mux.go","file.line":133},"message":"Path /v1/logs added to request handler","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server    | {"log.level":"info","@timestamp":"2022-11-27T20:26:32.296Z","log.logger":"beater","log.origin":{"file.name":"beater/server.go","file.line":193},"message":"Starting apm-server [603b5878ad5a02225c03970741b87fb7f8c38e4a built 2022-11-17 10:57:37 +0100 +0100]. Hit CTRL-C to stop it.","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server    | {"log.level":"info","@timestamp":"2022-11-27T20:26:32.296Z","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":94},"message":"RUM endpoints disabled.","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server    | {"log.level":"info","@timestamp":"2022-11-27T20:26:32.296Z","log.logger":"beater","log.origin":{"file.name":"beater/http.go","file.line":104},"message":"SSL disabled.","service.name":"apm-server","ecs.version":"1.6.0"}
apm-server    | {"log.level":"error","@timestamp":"2022-11-27T20:26:32.297Z","log.logger":"beater","log.origin":{"file.name":"beater/waitready.go","file.line":64},"message":"precondition failed: error querying cluster_uuid: status_code=401","service.name":"apm-server","ecs.version":"1.6.0"}

[stephenb (Stephen Brown)]

Perhaps look at the compose file here

Is this a brand new cluster?... I think you might need that the setup config etc

Having trouble replicating right now.. haven't had a chance to set up.

Changed: Question did you actually add the APM Integration you need to do that to / add the assets even if you are not using fleet .. that is where the templates get loaded etc.

[stephenb (Stephen Brown)]

Ok I got mine working...

I think you did not setup / load the APM integration... kibana weird but you still need to do that... you need to do that before you start the APM Server

Then I see the assets

GET _cat/indices/.*?v
GET _cat/templates/tra*

# GET _cat/indices/.*?v 200 OK
health status index                                                         uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   .fleet-enrollment-api-keys-7                                  0JXPxyXpRFarcWU-7QBuBw   1   0          1            0      6.1kb          6.1kb
green  open   .apm-agent-configuration                                      Y2n3cINATZyg9xhOVsJmmw   1   0          0            0       225b           225b
green  open   .kibana_security_session_1                                    dFOtc-OnTMqevxUGc9p5tw   1   0          1            0      5.7kb          5.7kb
green  open   .kibana-event-log-8.4.0-000001                                LWUfHiPDShyzT_POBu23SA   1   0          1            0      6.2kb          6.2kb
green  open   .geoip_databases                                              C_e4xhUBSQa5IrvzeBfUJQ   1   0         41            0     39.2mb         39.2mb
green  open   .security-7                                                   SxKjtS3wQy2V4UU-G3q5CA   1   0        111            0      336kb          336kb
green  open   .kibana_task_manager_8.4.0_001                                --xVYRnoSt2H4yKCNPD7vw   1   0         25         1397    238.1kb        238.1kb
green  open   .apm-custom-link                                              3lBvbBxQSgW6bd4PWKfiiQ   1   0          0            0       225b           225b
green  open   .kibana_8.4.0_001                                             jBOCGWxGQT-iTKg5OZ_QOA   1   0       2385         1226      8.9mb          8.9mb
green  open   .security-profile-8                                           NT5vs6sOQfW113PXPh0zvA   1   0          1            0      7.7kb          7.7kb
green  open   .fleet-policies-7                                             Sz0IgOoDSuepT3uvZBzEoQ   1   0          3            0       22kb           22kb
green  open   .ds-.logs-deprecation.elasticsearch-default-2022.11.28-000001 cQDV1sFnSyyrDGAOGImXNg   1   0          1            0     12.3kb         12.3kb
green  open   .ds-ilm-history-5-2022.11.28-000001                           W5owRm08QAmWpKbotz9DrQ   1   0          9            0     35.6kb         35.6kb

# GET _cat/templates/tra* 200 OK
traces-apm         [traces-apm-*]         200  [traces-apm@package, traces-apm@custom, .fleet_globals-1, .fleet_agent_id_verification-1]
traces-apm.sampled [traces-apm.sampled-*] 200  [traces-apm.sampled@package, traces-apm.sampled@custom, .fleet_globals-1, .fleet_agent_id_verification-1]
traces-apm.rum     [traces-apm.rum-*]     200  [traces-apm.rum@package, traces-apm.rum@custom, .fleet_globals-1, .fleet_agent_id_verification-1]

Then my apm-server.docker.yml looks like this

apm-server:
  # Defines the host and port the server is listening on. Use "unix:/path/to.sock" to listen on a unix domain socket.
  host: "0.0.0.0:8200"

#-------------------------- Elasticsearch output --------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  # Scheme and port can be left out and will be set to the default (`http` and `9200`).
  # In case you specify and additional path, the scheme is required: `http://elasticsearch:9200/path`.
  # IPv6 addresses should always be defined as: `https://[2001:db8::1]:9200`.
  hosts: ["https://host.docker.internal:9200"]
  ssl.verification_mode: none
  username: "elastic"
  password: "password"

Then I ran...

docker run -d \
  -p 8200:8200 \
  --name=apm-server \
  --user=apm-server \
  --volume="$(pwd)/apm-server.docker.yml:/usr/share/apm-server/apm-server.yml:ro" \
  docker.elastic.co/apm/apm-server:8.5.2 \
  --strict.perms=false -e

Then I got this