# Embedding Certificate in configuration file fails with environment variable

**URL:** <https://discuss.elastic.co/t/embedding-certificate-in-configuration-file-fails-with-environment-variable/347554>\
**Category:** APM\
**Tags:** elastic-stack-security, docker, server\
**Created:** [November 20, 2023, 9:03pm UTC](https://discuss.elastic.co/t/embedding-certificate-in-configuration-file-fails-with-environment-variable/347554 "2023-11-20T21:03:22Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![asnyameeteen](https://avatars.discourse-cdn.com/v4/letter/a/919ad9/32.png) [@asnyameeteen](https://discuss.elastic.co/u/asnyameeteen)\
**Post date:** [November 20, 2023, 9:03pm UTC](https://discuss.elastic.co/t/embedding-certificate-in-configuration-file-fails-with-environment-variable/347554/1 "2023-11-20T21:03:22Z")

</div>

Running versions 8.10.2 of Kibana, Filebeat, Elasticsearch and APM Server. They are currently running on AWS ECS as Docker containers. I am attempting to configure the `output.elasticsearch` portion of the `apm-server.yml` with environment variables like so:

```auto
output.elasticsearch:
  enabled: true
  protocol: "https"
  hosts:
    - ${ELASTICSEARCH_HOST}
  username: ${USERNAME}
  password: ${PASSWORD}
  ssl:
    enabled: true
    certificate_authorities:
      - |
        ${CA}
    certificate: |
      ${CERTIFICATE}
    key: |
      ${PRIVATE_KEY}

```

This is the Dockerfile:

```auto
FROM elastic/apm-server:8.10.2
COPY --chmod=0644 --chown=1000:1000 apm-server.yml /usr/share/apm-server/apm-server.yml

```

The CA loads correctly but the certificate fails with the following error:  
`Error: error unpacking output.elasticsearch for fetching agent config: can not convert 'object' into 'string' accessing 'output.elasticsearch.ssl.certificate' (source:'apm-server.yml')`  
Is this a bug or am I doing something wrong?

---

<div class="post-metadata">

**Author:** ![Carson\_Ip](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/carson_ip/32/115912_2.png) [@Carson\_Ip](https://discuss.elastic.co/u/Carson_Ip)\
**Post date:** [November 21, 2023, 5:47pm UTC](https://discuss.elastic.co/t/embedding-certificate-in-configuration-file-fails-with-environment-variable/347554/2 "2023-11-21T17:47:46Z")

</div>

Hi @asnyameeteen ,

I've tested a similar setup with CERTIFICATE and PRIVATE\_KEY in env var, and cannot reproduce your issue. It should be able to work. Can you provide more error logs if possible? Did the program terminate immediately? Also, can you double check if CERTIFICATE and PRIVATE\_KEY contain the correct content and is properly formatted with newlines?

---

<div class="post-metadata">

**Author:** ![asnyameeteen](https://avatars.discourse-cdn.com/v4/letter/a/919ad9/32.png) [@asnyameeteen](https://discuss.elastic.co/u/asnyameeteen)\
**Post date:** [November 21, 2023, 7:11pm UTC](https://discuss.elastic.co/t/embedding-certificate-in-configuration-file-fails-with-environment-variable/347554/3 "2023-11-21T19:11:17Z")

</div>

It does exit immediately. Here is the entire stack trace.

```auto
{"log.level":"info","@timestamp":"2023-11-21T19:02:38.664Z","log.origin":{"file.name":"beatcmd/beat.go","file.line":139},"message":"Home path: [/usr/share/apm-server] Config path: [/usr/share/apm-server] Data path: [/usr/share/apm-server/data] Logs path: [/usr/share/apm-server/logs]","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-21T19:02:38.668Z","log.origin":{"file.name":"beatcmd/beat.go","file.line":146},"message":"Beat ID: dd0023e6-383a-4407-8d1b-12ccec76263c","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-21T19:02:38.669Z","log.logger":"beat","log.origin":{"file.name":"beatcmd/beat.go","file.line":576},"message":"Beat info","service.name":"apm-server","system_info":{"beat":{"path":{"config":"/usr/share/apm-server","data":"/usr/share/apm-server/data","home":"/usr/share/apm-server","logs":"/usr/share/apm-server/logs"},"type":"apm-server","uuid":"dd0023e6-383a-4407-8d1b-12ccec76263c"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-11-21T19:02:38.669Z","log.logger":"beat","log.origin":{"file.name":"beatcmd/beat.go","file.line":584},"message":"Build info","service.name":"apm-server","system_info":{"build":{"commit":"825865682816d18b5e66e94949bce8d2f0172044","time":"2023-09-18T17:00:04.000Z","version":"8.10.2"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-11-21T19:02:38.669Z","log.logger":"beat","log.origin":{"file.name":"beatcmd/beat.go","file.line":587},"message":"Go runtime info","service.name":"apm-server","system_info":{"go":{"os":"linux","arch":"amd64","max_procs":2,"version":"go1.20.7"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-11-21T19:02:38.673Z","log.origin":{"file.name":"beatcmd/maxprocs.go","file.line":68},"message":"maxprocs: Leaving GOMAXPROCS=2: CPU quota undefined","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-21T19:02:38.677Z","log.logger":"beat","log.origin":{"file.name":"beatcmd/beat.go","file.line":591},"message":"Host info","service.name":"apm-server","system_info":{"host":{"architecture":"x86_64","boot_time":"2023-07-17T14:27:47Z","containerized":false,"name":"8d7f6afb7c56","ip":["127.0.0.1/8","172.17.0.8/16"],"kernel_version":"4.14.291-218.527.amzn2.x86_64","mac":["02:42:ac:11:00:08"],"os":{"type":"linux","family":"debian","platform":"ubuntu","name":"Ubuntu","version":"20.04.6 LTS (Focal Fossa)","major":20,"minor":4,"patch":6,"codename":"focal"},"timezone":"UTC","timezone_offset_sec":0},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-11-21T19:02:38.677Z","log.logger":"beat","log.origin":{"file.name":"beatcmd/beat.go","file.line":620},"message":"Process info","service.name":"apm-server","system_info":{"process":{"capabilities":{"inheritable":null,"permitted":null,"effective":null,"bounding":["chown","dac_override","fowner","fsetid","kill","setgid","setuid","setpcap","net_bind_service","net_raw","sys_chroot","mknod","audit_write","setfcap"],"ambient":null},"cwd":"/usr/share/apm-server","exe":"/usr/share/apm-server/apm-server","name":"apm-server","pid":8,"ppid":1,"seccomp":{"mode":"filter","no_new_privs":false},"start_time":"2023-11-21T19:02:37.270Z"},"ecs.version":"1.6.0"}}
{"log.level":"info","@timestamp":"2023-11-21T19:02:38.680Z","log.logger":"config","log.origin":{"file.name":"config/agentconfig.go","file.line":70},"message":"using output.elasticsearch for fetching agent config","service.name":"apm-server","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2023-11-21T19:02:38.683Z","log.origin":{"file.name":"beatcmd/beat.go","file.line":390},"message":"apm-server stopped.","service.name":"apm-server","ecs.version":"1.6.0"}
Error: error unpacking output.elasticsearch for fetching agent config: can not convert 'object' into 'string' accessing 'output.elasticsearch.ssl.certificate' (source:'apm-server.yml')
Usage:
  apm-server [flags]
  apm-server [command]

Available Commands:
  apikey Manage API Keys for communication between APM agents and server (deprecated)
  export Export current config
  help Help about any command
  keystore Manage secrets keystore
  run Run APM Server
  test Test config
  version Show current version info

Flags:
  -E, --E setting=value Configuration overwrite
  -N, --N Disable actual publishing for testing
  -c, --c string Configuration file, relative to path.config (default "apm-server.yml")
      --cpuprofile string Write cpu profile to file
  -d, --d stringArray Enable certain debug selectors
  -e, --e Log to stderr and disable syslog/file output
      --environment string Set the environment in which the process is running (default "default")
  -h, --help help for apm-server
      --httpprof string Start pprof http server
      --memprofile string Write memory profile to this file
      --path.config string Configuration path
      --path.data string Data path
      --path.home string Home path
      --path.logs string Logs path
      --strict.perms Strict permission checking on config files (default true)
  -v, --v Log at INFO level

Use "apm-server [command] --help" for more information about a command.

```

The AWS task definition uses secrets from Parameter Store saved as SecureString type. The CA, the CERTIFICATE and the PRIVATE\_KEY variables are injected by ECS from Parameter Store. When I run the `docker inspect` command on the dead container, the values are as expected.

---

<div class="post-metadata">

**Author:** ![asnyameeteen](https://avatars.discourse-cdn.com/v4/letter/a/919ad9/32.png) [@asnyameeteen](https://discuss.elastic.co/u/asnyameeteen)\
**Post date:** [December 6, 2023, 11:32pm UTC](https://discuss.elastic.co/t/embedding-certificate-in-configuration-file-fails-with-environment-variable/347554/4 "2023-12-06T23:32:49Z")

</div>

I am still not able to successfully get the CA embedded using AWS Systems Manager Parameter Store

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 3, 2024, 11:33pm UTC](https://discuss.elastic.co/t/embedding-certificate-in-configuration-file-fails-with-environment-variable/347554/5 "2024-01-03T23:33:12Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.