Embedding Certificate in configuration file fails with environment variable

Running versions 8.10.2 of Kibana, Filebeat, Elasticsearch and APM Server. They are currently running on AWS ECS as Docker containers. I am attempting to configure the output.elasticsearch portion of the apm-server.yml with environment variables like so:

output.elasticsearch:
  enabled: true
  protocol: "https"
  hosts:
    - ${ELASTICSEARCH_HOST}
  username: ${USERNAME}
  password: ${PASSWORD}
  ssl:
    enabled: true
    certificate_authorities:
      - |
        ${CA}
    certificate: |
      ${CERTIFICATE}
    key: |
      ${PRIVATE_KEY}

This is the Dockerfile:

FROM elastic/apm-server:8.10.2
COPY --chmod=0644 --chown=1000:1000 apm-server.yml /usr/share/apm-server/apm-server.yml

The CA loads correctly but the certificate fails with the following error:
Error: error unpacking output.elasticsearch for fetching agent config: can not convert 'object' into 'string' accessing 'output.elasticsearch.ssl.certificate' (source:'apm-server.yml')
Is this a bug or am I doing something wrong?

Hi @asnyameeteen ,

I've tested a similar setup with CERTIFICATE and PRIVATE_KEY in env var, and cannot reproduce your issue. It should be able to work. Can you provide more error logs if possible? Did the program terminate immediately? Also, can you double check if CERTIFICATE and PRIVATE_KEY contain the correct content and is properly formatted with newlines?

It does exit immediately. Here is the entire stack trace.

{"log.level":"info","@timestamp":"2023-11-21T19:02:38.664Z","log.origin":{"file.name":"beatcmd/beat.go","file.line":139},"message":"Home path: [/usr/share/apm-server] Config path: [/usr/share/apm-server] Data path: [/usr/share/apm-server/data] Logs path: [/usr/share/apm-server/logs]","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-21T19:02:38.668Z","log.origin":{"file.name":"beatcmd/beat.go","file.line":146},"message":"Beat ID: dd0023e6-383a-4407-8d1b-12ccec76263c","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-21T19:02:38.669Z","log.logger":"beat","log.origin":{"file.name":"beatcmd/beat.go","file.line":576},"message":"Beat info","service.name":"apm-server","system_info":{"beat":{"path":{"config":"/usr/share/apm-server","data":"/usr/share/apm-server/data","home":"/usr/share/apm-server","logs":"/usr/share/apm-server/logs"},"type":"apm-server","uuid":"dd0023e6-383a-4407-8d1b-12ccec76263c"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-11-21T19:02:38.669Z","log.logger":"beat","log.origin":{"file.name":"beatcmd/beat.go","file.line":584},"message":"Build info","service.name":"apm-server","system_info":{"build":{"commit":"825865682816d18b5e66e94949bce8d2f0172044","time":"2023-09-18T17:00:04.000Z","version":"8.10.2"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-11-21T19:02:38.669Z","log.logger":"beat","log.origin":{"file.name":"beatcmd/beat.go","file.line":587},"message":"Go runtime info","service.name":"apm-server","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":2,"version":"go1.20.7"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-11-21T19:02:38.673Z","log.origin":{"file.name":"beatcmd/maxprocs.go","file.line":68},"message":"maxprocs: Leaving GOMAXPROCS=2: CPU quota undefined","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-21T19:02:38.677Z","log.logger":"beat","log.origin":{"file.name":"beatcmd/beat.go","file.line":591},"message":"Host info","service.name":"apm-server","system_info":{"host":{"architecture":"x86_64","boot_time":"2023-07-17T14:27:47Z","containerized":false,"name":"8d7f6afb7c56","ip":["127.0.0.1/8","172.17.0.8/16"],"kernel_version":"4.14.291-218.527.amzn2.x86_64","mac":["02:42:ac:11:00:08"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"20.04.6 LTS (Focal Fossa)","major":20,"minor":4,"patch":6,"codename":"focal"},"timezone":"UTC","timezone_offset_sec":0},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-11-21T19:02:38.677Z","log.logger":"beat","log.origin":{"file.name":"beatcmd/beat.go","file.line":620},"message":"Process info","service.name":"apm-server","system_info":{"process":{"capabilities":{"inheritable":null,"permitted":null,"effective":null,"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null},"cwd":"/usr/share/apm-server","exe":"/usr/share/apm-server/apm-server","name":"apm-server","pid":8,"ppid":1,"seccomp":{"mode":"filter","no_new_privs":false},"start_time":"2023-11-21T19:02:37.270Z"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-11-21T19:02:38.680Z","log.logger":"config","log.origin":{"file.name":"config/agentconfig.go","file.line":70},"message":"using output.elasticsearch for fetching agent config","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-21T19:02:38.683Z","log.origin":{"file.name":"beatcmd/beat.go","file.line":390},"message":"apm-server stopped.","service.name":"apm-server","ecs.version":"1.6.0"}
Error: error unpacking output.elasticsearch for fetching agent config: can not convert 'object' into 'string' accessing 'output.elasticsearch.ssl.certificate' (source:'apm-server.yml')
Usage:
  apm-server [flags]
  apm-server [command]

Available Commands:
  apikey      Manage API Keys for communication between APM agents and server (deprecated)
  export      Export current config
  help        Help about any command
  keystore    Manage secrets keystore
  run         Run APM Server
  test        Test config
  version     Show current version info

Flags:
  -E, --E setting=value      Configuration overwrite
  -N, --N                    Disable actual publishing for testing
  -c, --c string             Configuration file, relative to path.config (default "apm-server.yml")
      --cpuprofile string    Write cpu profile to file
  -d, --d stringArray        Enable certain debug selectors
  -e, --e                    Log to stderr and disable syslog/file output
      --environment string   Set the environment in which the process is running (default "default")
  -h, --help                 help for apm-server
      --httpprof string      Start pprof http server
      --memprofile string    Write memory profile to this file
      --path.config string   Configuration path
      --path.data string     Data path
      --path.home string     Home path
      --path.logs string     Logs path
      --strict.perms         Strict permission checking on config files (default true)
  -v, --v                    Log at INFO level

Use "apm-server [command] --help" for more information about a command.

The AWS task definition uses secrets from Parameter Store saved as SecureString type. The CA, the CERTIFICATE and the PRIVATE_KEY variables are injected by ECS from Parameter Store. When I run the docker inspect command on the dead container, the values are as expected.

I am still not able to successfully get the CA embedded using AWS Systems Manager Parameter Store

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.