Fleet Server - * missing enrollment api key

Hey All,

Recently upgraded to 7.14.1 to and am struggling to install fleet server. I am running on-prem and have my self-signed certs setup. Trying to install fleet-server on the same node where elasticsearch is installed. The IP of this server is 192.168.117.101, my Kibana instance is 192.168.117.11

From the logs, I'm guessing it a problem with a certificate and being able to access Kibana/Elasticsearch to get an enrollment key? Not too sure where I can start looking since I have followed all the documentation: Encrypt traffic in a self-managed cluster | Fleet User Guide [7.14] | Elastic

I am installing it via this command

sudo /root/elastic-agent/elastic-agent enroll -f --insecure \
  --url=https://192.168.117.101:8220 \
  --fleet-server-es=https://192.168.117.101:9200 \
  --fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE2MzA3NDA3NTM5ODQ6OTFkRm1FdXBRbXlwLWk4R3Nyd1p0UQ \
  --fleet-server-policy=0d8b9ba0-0ce5-11ec-8978-2bc18b6d481d \
  --certificate-authorities=/etc/elasticsearch/other/ca/ca.crt \
  --fleet-server-es-ca=/etc/elasticsearch/other/elasticsearch-ca.crt \
  --fleet-server-cert=/etc/elasticsearch/other/fleet-server/fleet-server.crt \
  --fleet-server-cert-key=/etc/elasticsearch/other/fleet-server/fleet-server.key

The output from the console is

2021-09-04T18:56:37.214+1000    INFO    cmd/enroll_cmd.go:508   Spawning Elastic Agent daemon as a subprocess to complete bootstrap process.
2021-09-04T18:56:37.337+1000    INFO    application/application.go:66   Detecting execution mode
2021-09-04T18:56:37.337+1000    INFO    application/application.go:87   Agent is in Fleet Server bootstrap mode
2021-09-04T18:56:37.581+1000    INFO    [api]   api/server.go:62        Starting stats endpoint
2021-09-04T18:56:37.581+1000    INFO    application/fleet_server_bootstrap.go:124       Agent is starting
2021-09-04T18:56:37.581+1000    INFO    [api]   api/server.go:64        Metrics endpoint listening on: /root/elastic-agent/data/tmp/elastic-agent.sock (configured: unix:///root/elastic-agent/data/tmp/elastic-agent.sock)
2021-09-04T18:56:37.581+1000    INFO    application/fleet_server_bootstrap.go:134       Agent is stopped
2021-09-04T18:56:37.582+1000    INFO    stateresolver/stateresolver.go:48       New State ID is OUJQIKiU
2021-09-04T18:56:37.582+1000    INFO    stateresolver/stateresolver.go:49       Converging state requires execution of 1 step(s)
2021-09-04T18:56:37.603+1000    INFO    operation/operator.go:260       operation 'operation-install' skipped for fleet-server.7.14.1
2021-09-04T18:56:37.738+1000    INFO    log/reporter.go:40      2021-09-04T18:56:37+10:00 - message: Application: fleet-server--7.14.1[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-09-04T18:56:37.739+1000    INFO    stateresolver/stateresolver.go:66       Updating internal state
2021-09-04T18:56:38.215+1000    INFO    cmd/enroll_cmd.go:683   Fleet Server - Starting
2021-09-04T18:56:38.254+1000    INFO    log/reporter.go:40      2021-09-04T18:56:38+10:00 - message: Application: fleet-server--7.14.1[]: State changed to RESTARTING: exited with code: 1 - type: 'STATE' - sub_type: 'STARTING'
2021-09-04T18:56:38.254+1000    INFO    log/reporter.go:40      2021-09-04T18:56:38+10:00 - message: Application: fleet-server--7.14.1[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-09-04T18:56:38.254+1000    INFO    log/reporter.go:40      2021-09-04T18:56:38+10:00 - message: Application: fleet-server--7.14.1[]: State changed to RESTARTING: Restarting - type: 'STATE' - sub_type: 'STARTING'
2021-09-04T18:56:38.767+1000    INFO    log/reporter.go:40      2021-09-04T18:56:38+10:00 - message: Application: fleet-server--7.14.1[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-09-04T18:56:44.219+1000    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-04T18:56:50.224+1000    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-04T18:56:56.229+1000    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-04T18:57:02.234+1000    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-04T18:57:08.238+1000    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-04T18:57:14.242+1000    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-04T18:57:20.248+1000    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-04T18:57:26.253+1000    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-04T18:57:32.258+1000    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-04T18:57:38.272+1000    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-04T18:57:42.589+1000    WARN    status/reporter.go:236  Elastic Agent status changed to: 'degraded'
2021-09-04T18:57:42.589+1000    INFO    log/reporter.go:40      2021-09-04T18:57:42+10:00 - message: Application: fleet-server--7.14.1[]: State changed to DEGRADED: Missed last check-in - type: 'STATE' - sub_type: 'RUNNING'
2021-09-04T18:57:43.276+1000    INFO    cmd/enroll_cmd.go:664   Fleet Server - Missed last check-in
2021-09-04T18:57:43.276+1000    WARN    [tls]   tlscommon/tls_config.go:98      SSL/TLS verifications disabled.
2021-09-04T18:57:43.838+1000    INFO    cmd/enroll_cmd.go:396   Starting enrollment to URL: https://192.168.117.101:8220/
2021-09-04T18:57:43.939+1000    INFO    cmd/run.go:189  Shutting down Elastic Agent and sending last events...
2021-09-04T18:57:43.939+1000    INFO    operation/operator.go:192       waiting for installer of pipeline 'default' to finish
2021-09-04T18:57:43.939+1000    INFO    process/app.go:176      Signaling application to stop because of shutdown: fleet-server--7.14.1
2021-09-04T18:58:14.978+1000    INFO    status/reporter.go:236  Elastic Agent status changed to: 'online'
2021-09-04T18:58:14.978+1000    INFO    cmd/run.go:197  Shutting down completed.
2021-09-04T18:58:14.978+1000    INFO    log/reporter.go:40      2021-09-04T18:58:14+10:00 - message: Application: fleet-server--7.14.1[]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'
2021-09-04T18:58:14.978+1000    INFO    [api]   api/server.go:66        Stats endpoint (/root/elastic-agent/data/tmp/elastic-agent.sock) finished: accept unix /root/elastic-agent/data/tmp/elastic-agent.sock: use of closed network connection
Error: fail to enroll: fail to execute request to fleet-server: 1 error occurred:
        * missing enrollment api key

Fleet-Server logs

{"log.level":"info","service.name":"fleet-server","version":"7.14.1","commit":"834362b","pid":2618391,"ppid":2618352,"exe":"/root/elastic-agent/data/elastic-agent-703d58/install/fleet-server-7.14.1-linux-x86_64/fleet-server","args":["--agent-mode","-E","logging.level=info","-E","http.enabled=true","-E","http.host=unix:///root/elastic-agent/data/tmp/default/fleet-server/fleet-server.sock","-E","logging.json=true","-E","logging.ecs=true","-E","logging.files.path=/root/elastic-agent/data/elastic-agent-703d58/logs/default","-E","logging.files.name=fleet-server-json.log","-E","logging.files.keepfiles=7","-E","logging.files.permission=0640","-E","logging.files.interval=1h","-E","path.data=/root/elastic-agent/data/elastic-agent-703d58/run/default/fleet-server--7.14.1"],"@timestamp":"2021-09-04T08:56:38.262Z","message":"boot"}
{"log.level":"info","service.name":"fleet-server","@timestamp":"2021-09-04T08:56:38.262Z","message":"starting communication connection back to Elastic Agent"}
{"log.level":"info","service.name":"fleet-server","@timestamp":"2021-09-04T08:56:38.262Z","message":"waiting for Elastic Agent to send initial configuration"}
{"log.level":"error","service.name":"fleet-server","error.message":"1 error: file is not a certificate adding /etc/elasticsearch/other/elasticsearch-ca.crt to the list of known CAs accessing 'output.elasticsearch'","@timestamp":"2021-09-04T08:56:38.768Z","message":"Exiting"}
error  indicates a PEM file to be loaded not being a valid PEM file or certificate.

can you double check on that?

@Michal_Pristas, I am also getting "missing enrollment api key" when trying to install fleet server 7.14.1 on the same node where elasticsearch and kibana are installed.

D:\elastic-agent-7.14.1-windows-x86_64>.\elastic-agent.exe install -f --fleet-server-es=https://localhost:9200 --fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE2MzIyMjgzMDkyODc6eXQyTWk5cUVSMlM1dVlhaXRFaVJpUQ --fleet-server-policy=6bcf1aa0-0fa8-11ec-bd0d-ad88f2617109 --fleet-server-es-ca=D:\elasticsearch-7.14.1\elastic-ca.p12
2021-09-21T19:23:59.970+0545    INFO    cmd/enroll_cmd.go:336   Generating self-signed certificate for Fleet Server
2021-09-21T19:24:02.327+0545    INFO    cmd/enroll_cmd.go:650   Waiting for Elastic Agent to start Fleet Server
2021-09-21T19:24:04.379+0545    INFO    cmd/enroll_cmd.go:633   Waiting for Elastic Agent to start
2021-09-21T19:24:06.407+0545    INFO    cmd/enroll_cmd.go:683   Fleet Server - Starting
2021-09-21T19:24:08.458+0545    INFO    cmd/enroll_cmd.go:683   Fleet Server - Restarting
2021-09-21T19:24:09.473+0545    INFO    cmd/enroll_cmd.go:683   Fleet Server - Starting
2021-09-21T19:24:15.636+0545    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-21T19:24:21.717+0545    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-21T19:24:27.800+0545    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-21T19:24:33.905+0545    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-21T19:24:39.976+0545    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-21T19:24:46.047+0545    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-21T19:24:52.127+0545    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-21T19:24:58.256+0545    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-21T19:25:04.363+0545    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-21T19:25:10.583+0545    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-21T19:25:11.595+0545    INFO    cmd/enroll_cmd.go:664   Fleet Server - Missed last check-in
2021-09-21T19:25:11.655+0545    INFO    cmd/enroll_cmd.go:396   Starting enrollment to URL: https://WIN10E:8220/
Error: fail to enroll: fail to execute request to fleet-server: 1 error occurred:
        * missing enrollment api key

Please look into this matter.

Have you tried removing the agent completely, installing the elastic agent and then enrolling the policy with the Fleet server ?

This worked for me after Endpoint Security screwed up my installation

I was following their guide on Add a Fleet Server.

Do you mean to start again with creating a default Fleet Server policy?

If it is not an issue, it is an easier way to say wasting time troubleshooting a problem that might be a needle in a haystack.

Did everything again. Just ran into different errors.
Bottom line - The --insecure flag does not work in latest versions.

Hey Vergil,

So I re-issued all my certificates (I had to since they were expiring in 6 months) and it worked. I didn't need to use the insecure flag. What I also had to add was a "--fleet-server-service-token". This is the specific command I ran to enroll it - I have blocked our all the confidential parts.

/root/elastic-agent/elastic-agent enroll --url=https://192.168.0.2:8220 \
  -f \
 --fleet-server-es=https://192.168.0.2:9200 \
 --fleet-server-service-token=THIS_WAS_90_CHARACTERS_OF_CAPITALS_LOWERCASE_AND_NUMBERS \
 --fleet-server-policy=THISWAS-NUMBERS-AND-CHARACTERS- \
 --certificate-authorities=/root/elastic-agent/ca.crt \
 --fleet-server-es-ca=/root/elastic-agent/ca.crt \
 --fleet-server-cert=/root/elastic-agent/fleet-server.crt \
 --fleet-server-cert-key=/root/elastic-agent/fleet-server.key 

After running that, I was able to start it with

systemctl start elastic-agent

You can also check the status - which I found useful by typing

elastic-agent status

My cluster also started around version 7.4, so I had Beat Management (the management thing prior to fleet-server). I had to delete various API tokens before some Fleet-Server features came up.

So, you just re-issued all your certificates and it worked.

Yep. Probably didn’t do my certificates correctly the first time (didn’t have much SSL knowledge)

Is your fleet server CA different from your elastic CA?

Maybe try this for the fleet server config within Kibana. I also had to copy the certificate into the output part.

Section 1 e. Under “Configure Fleet settings”

https://www.elastic.co/guide/en/fleet/current/secure-connections.html

Don't worry. My issue got solved just by again creating certs for the fleet server.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.