Fleet Server - * missing enrollment api key

Hey All,

Recently upgraded to 7.14.1 to and am struggling to install fleet server. I am running on-prem and have my self-signed certs setup. Trying to install fleet-server on the same node where elasticsearch is installed. The IP of this server is 192.168.117.101, my Kibana instance is 192.168.117.11

From the logs, I'm guessing it a problem with a certificate and being able to access Kibana/Elasticsearch to get an enrollment key? Not too sure where I can start looking since I have followed all the documentation: Encrypt traffic in a self-managed cluster | Fleet User Guide [7.14] | Elastic

I am installing it via this command

sudo /root/elastic-agent/elastic-agent enroll -f --insecure \
  --url=https://192.168.117.101:8220 \
  --fleet-server-es=https://192.168.117.101:9200 \
  --fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE2MzA3NDA3NTM5ODQ6OTFkRm1FdXBRbXlwLWk4R3Nyd1p0UQ \
  --fleet-server-policy=0d8b9ba0-0ce5-11ec-8978-2bc18b6d481d \
  --certificate-authorities=/etc/elasticsearch/other/ca/ca.crt \
  --fleet-server-es-ca=/etc/elasticsearch/other/elasticsearch-ca.crt \
  --fleet-server-cert=/etc/elasticsearch/other/fleet-server/fleet-server.crt \
  --fleet-server-cert-key=/etc/elasticsearch/other/fleet-server/fleet-server.key

The output from the console is

2021-09-04T18:56:37.214+1000    INFO    cmd/enroll_cmd.go:508   Spawning Elastic Agent daemon as a subprocess to complete bootstrap process.
2021-09-04T18:56:37.337+1000    INFO    application/application.go:66   Detecting execution mode
2021-09-04T18:56:37.337+1000    INFO    application/application.go:87   Agent is in Fleet Server bootstrap mode
2021-09-04T18:56:37.581+1000    INFO    [api]   api/server.go:62        Starting stats endpoint
2021-09-04T18:56:37.581+1000    INFO    application/fleet_server_bootstrap.go:124       Agent is starting
2021-09-04T18:56:37.581+1000    INFO    [api]   api/server.go:64        Metrics endpoint listening on: /root/elastic-agent/data/tmp/elastic-agent.sock (configured: unix:///root/elastic-agent/data/tmp/elastic-agent.sock)
2021-09-04T18:56:37.581+1000    INFO    application/fleet_server_bootstrap.go:134       Agent is stopped
2021-09-04T18:56:37.582+1000    INFO    stateresolver/stateresolver.go:48       New State ID is OUJQIKiU
2021-09-04T18:56:37.582+1000    INFO    stateresolver/stateresolver.go:49       Converging state requires execution of 1 step(s)
2021-09-04T18:56:37.603+1000    INFO    operation/operator.go:260       operation 'operation-install' skipped for fleet-server.7.14.1
2021-09-04T18:56:37.738+1000    INFO    log/reporter.go:40      2021-09-04T18:56:37+10:00 - message: Application: fleet-server--7.14.1[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-09-04T18:56:37.739+1000    INFO    stateresolver/stateresolver.go:66       Updating internal state
2021-09-04T18:56:38.215+1000    INFO    cmd/enroll_cmd.go:683   Fleet Server - Starting
2021-09-04T18:56:38.254+1000    INFO    log/reporter.go:40      2021-09-04T18:56:38+10:00 - message: Application: fleet-server--7.14.1[]: State changed to RESTARTING: exited with code: 1 - type: 'STATE' - sub_type: 'STARTING'
2021-09-04T18:56:38.254+1000    INFO    log/reporter.go:40      2021-09-04T18:56:38+10:00 - message: Application: fleet-server--7.14.1[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-09-04T18:56:38.254+1000    INFO    log/reporter.go:40      2021-09-04T18:56:38+10:00 - message: Application: fleet-server--7.14.1[]: State changed to RESTARTING: Restarting - type: 'STATE' - sub_type: 'STARTING'
2021-09-04T18:56:38.767+1000    INFO    log/reporter.go:40      2021-09-04T18:56:38+10:00 - message: Application: fleet-server--7.14.1[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-09-04T18:56:44.219+1000    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-04T18:56:50.224+1000    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-04T18:56:56.229+1000    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-04T18:57:02.234+1000    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-04T18:57:08.238+1000    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-04T18:57:14.242+1000    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-04T18:57:20.248+1000    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-04T18:57:26.253+1000    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-04T18:57:32.258+1000    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-04T18:57:38.272+1000    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-04T18:57:42.589+1000    WARN    status/reporter.go:236  Elastic Agent status changed to: 'degraded'
2021-09-04T18:57:42.589+1000    INFO    log/reporter.go:40      2021-09-04T18:57:42+10:00 - message: Application: fleet-server--7.14.1[]: State changed to DEGRADED: Missed last check-in - type: 'STATE' - sub_type: 'RUNNING'
2021-09-04T18:57:43.276+1000    INFO    cmd/enroll_cmd.go:664   Fleet Server - Missed last check-in
2021-09-04T18:57:43.276+1000    WARN    [tls]   tlscommon/tls_config.go:98      SSL/TLS verifications disabled.
2021-09-04T18:57:43.838+1000    INFO    cmd/enroll_cmd.go:396   Starting enrollment to URL: https://192.168.117.101:8220/
2021-09-04T18:57:43.939+1000    INFO    cmd/run.go:189  Shutting down Elastic Agent and sending last events...
2021-09-04T18:57:43.939+1000    INFO    operation/operator.go:192       waiting for installer of pipeline 'default' to finish
2021-09-04T18:57:43.939+1000    INFO    process/app.go:176      Signaling application to stop because of shutdown: fleet-server--7.14.1
2021-09-04T18:58:14.978+1000    INFO    status/reporter.go:236  Elastic Agent status changed to: 'online'
2021-09-04T18:58:14.978+1000    INFO    cmd/run.go:197  Shutting down completed.
2021-09-04T18:58:14.978+1000    INFO    log/reporter.go:40      2021-09-04T18:58:14+10:00 - message: Application: fleet-server--7.14.1[]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'
2021-09-04T18:58:14.978+1000    INFO    [api]   api/server.go:66        Stats endpoint (/root/elastic-agent/data/tmp/elastic-agent.sock) finished: accept unix /root/elastic-agent/data/tmp/elastic-agent.sock: use of closed network connection
Error: fail to enroll: fail to execute request to fleet-server: 1 error occurred:
        * missing enrollment api key

Fleet-Server logs

{"log.level":"info","service.name":"fleet-server","version":"7.14.1","commit":"834362b","pid":2618391,"ppid":2618352,"exe":"/root/elastic-agent/data/elastic-agent-703d58/install/fleet-server-7.14.1-linux-x86_64/fleet-server","args":["--agent-mode","-E","logging.level=info","-E","http.enabled=true","-E","http.host=unix:///root/elastic-agent/data/tmp/default/fleet-server/fleet-server.sock","-E","logging.json=true","-E","logging.ecs=true","-E","logging.files.path=/root/elastic-agent/data/elastic-agent-703d58/logs/default","-E","logging.files.name=fleet-server-json.log","-E","logging.files.keepfiles=7","-E","logging.files.permission=0640","-E","logging.files.interval=1h","-E","path.data=/root/elastic-agent/data/elastic-agent-703d58/run/default/fleet-server--7.14.1"],"@timestamp":"2021-09-04T08:56:38.262Z","message":"boot"}
{"log.level":"info","service.name":"fleet-server","@timestamp":"2021-09-04T08:56:38.262Z","message":"starting communication connection back to Elastic Agent"}
{"log.level":"info","service.name":"fleet-server","@timestamp":"2021-09-04T08:56:38.262Z","message":"waiting for Elastic Agent to send initial configuration"}
{"log.level":"error","service.name":"fleet-server","error.message":"1 error: file is not a certificate adding /etc/elasticsearch/other/elasticsearch-ca.crt to the list of known CAs accessing 'output.elasticsearch'","@timestamp":"2021-09-04T08:56:38.768Z","message":"Exiting"}
error  indicates a PEM file to be loaded not being a valid PEM file or certificate.

can you double check on that?