Fleet Server - * missing enrollment api key

bevano · September 4, 2021, 9:06am

Hey All,

Recently upgraded to 7.14.1 to and am struggling to install fleet server. I am running on-prem and have my self-signed certs setup. Trying to install fleet-server on the same node where elasticsearch is installed. The IP of this server is 192.168.117.101, my Kibana instance is 192.168.117.11

From the logs, I'm guessing it a problem with a certificate and being able to access Kibana/Elasticsearch to get an enrollment key? Not too sure where I can start looking since I have followed all the documentation: Encrypt traffic in a self-managed cluster | Fleet User Guide [7.14] | Elastic

I am installing it via this command

sudo /root/elastic-agent/elastic-agent enroll -f --insecure \
  --url=https://192.168.117.101:8220 \
  --fleet-server-es=https://192.168.117.101:9200 \
  --fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE2MzA3NDA3NTM5ODQ6OTFkRm1FdXBRbXlwLWk4R3Nyd1p0UQ \
  --fleet-server-policy=0d8b9ba0-0ce5-11ec-8978-2bc18b6d481d \
  --certificate-authorities=/etc/elasticsearch/other/ca/ca.crt \
  --fleet-server-es-ca=/etc/elasticsearch/other/elasticsearch-ca.crt \
  --fleet-server-cert=/etc/elasticsearch/other/fleet-server/fleet-server.crt \
  --fleet-server-cert-key=/etc/elasticsearch/other/fleet-server/fleet-server.key

The output from the console is

2021-09-04T18:56:37.214+1000    INFO    cmd/enroll_cmd.go:508   Spawning Elastic Agent daemon as a subprocess to complete bootstrap process.
2021-09-04T18:56:37.337+1000    INFO    application/application.go:66   Detecting execution mode
2021-09-04T18:56:37.337+1000    INFO    application/application.go:87   Agent is in Fleet Server bootstrap mode
2021-09-04T18:56:37.581+1000    INFO    [api]   api/server.go:62        Starting stats endpoint
2021-09-04T18:56:37.581+1000    INFO    application/fleet_server_bootstrap.go:124       Agent is starting
2021-09-04T18:56:37.581+1000    INFO    [api]   api/server.go:64        Metrics endpoint listening on: /root/elastic-agent/data/tmp/elastic-agent.sock (configured: unix:///root/elastic-agent/data/tmp/elastic-agent.sock)
2021-09-04T18:56:37.581+1000    INFO    application/fleet_server_bootstrap.go:134       Agent is stopped
2021-09-04T18:56:37.582+1000    INFO    stateresolver/stateresolver.go:48       New State ID is OUJQIKiU
2021-09-04T18:56:37.582+1000    INFO    stateresolver/stateresolver.go:49       Converging state requires execution of 1 step(s)
2021-09-04T18:56:37.603+1000    INFO    operation/operator.go:260       operation 'operation-install' skipped for fleet-server.7.14.1
2021-09-04T18:56:37.738+1000    INFO    log/reporter.go:40      2021-09-04T18:56:37+10:00 - message: Application: fleet-server--7.14.1[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-09-04T18:56:37.739+1000    INFO    stateresolver/stateresolver.go:66       Updating internal state
2021-09-04T18:56:38.215+1000    INFO    cmd/enroll_cmd.go:683   Fleet Server - Starting
2021-09-04T18:56:38.254+1000    INFO    log/reporter.go:40      2021-09-04T18:56:38+10:00 - message: Application: fleet-server--7.14.1[]: State changed to RESTARTING: exited with code: 1 - type: 'STATE' - sub_type: 'STARTING'
2021-09-04T18:56:38.254+1000    INFO    log/reporter.go:40      2021-09-04T18:56:38+10:00 - message: Application: fleet-server--7.14.1[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-09-04T18:56:38.254+1000    INFO    log/reporter.go:40      2021-09-04T18:56:38+10:00 - message: Application: fleet-server--7.14.1[]: State changed to RESTARTING: Restarting - type: 'STATE' - sub_type: 'STARTING'
2021-09-04T18:56:38.767+1000    INFO    log/reporter.go:40      2021-09-04T18:56:38+10:00 - message: Application: fleet-server--7.14.1[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-09-04T18:56:44.219+1000    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-04T18:56:50.224+1000    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-04T18:56:56.229+1000    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-04T18:57:02.234+1000    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-04T18:57:08.238+1000    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-04T18:57:14.242+1000    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-04T18:57:20.248+1000    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-04T18:57:26.253+1000    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-04T18:57:32.258+1000    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-04T18:57:38.272+1000    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-04T18:57:42.589+1000    WARN    status/reporter.go:236  Elastic Agent status changed to: 'degraded'
2021-09-04T18:57:42.589+1000    INFO    log/reporter.go:40      2021-09-04T18:57:42+10:00 - message: Application: fleet-server--7.14.1[]: State changed to DEGRADED: Missed last check-in - type: 'STATE' - sub_type: 'RUNNING'
2021-09-04T18:57:43.276+1000    INFO    cmd/enroll_cmd.go:664   Fleet Server - Missed last check-in
2021-09-04T18:57:43.276+1000    WARN    [tls]   tlscommon/tls_config.go:98      SSL/TLS verifications disabled.
2021-09-04T18:57:43.838+1000    INFO    cmd/enroll_cmd.go:396   Starting enrollment to URL: https://192.168.117.101:8220/
2021-09-04T18:57:43.939+1000    INFO    cmd/run.go:189  Shutting down Elastic Agent and sending last events...
2021-09-04T18:57:43.939+1000    INFO    operation/operator.go:192       waiting for installer of pipeline 'default' to finish
2021-09-04T18:57:43.939+1000    INFO    process/app.go:176      Signaling application to stop because of shutdown: fleet-server--7.14.1
2021-09-04T18:58:14.978+1000    INFO    status/reporter.go:236  Elastic Agent status changed to: 'online'
2021-09-04T18:58:14.978+1000    INFO    cmd/run.go:197  Shutting down completed.
2021-09-04T18:58:14.978+1000    INFO    log/reporter.go:40      2021-09-04T18:58:14+10:00 - message: Application: fleet-server--7.14.1[]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'
2021-09-04T18:58:14.978+1000    INFO    [api]   api/server.go:66        Stats endpoint (/root/elastic-agent/data/tmp/elastic-agent.sock) finished: accept unix /root/elastic-agent/data/tmp/elastic-agent.sock: use of closed network connection
Error: fail to enroll: fail to execute request to fleet-server: 1 error occurred:
        * missing enrollment api key

Fleet-Server logs

{"log.level":"info","service.name":"fleet-server","version":"7.14.1","commit":"834362b","pid":2618391,"ppid":2618352,"exe":"/root/elastic-agent/data/elastic-agent-703d58/install/fleet-server-7.14.1-linux-x86_64/fleet-server","args":["--agent-mode","-E","logging.level=info","-E","http.enabled=true","-E","http.host=unix:///root/elastic-agent/data/tmp/default/fleet-server/fleet-server.sock","-E","logging.json=true","-E","logging.ecs=true","-E","logging.files.path=/root/elastic-agent/data/elastic-agent-703d58/logs/default","-E","logging.files.name=fleet-server-json.log","-E","logging.files.keepfiles=7","-E","logging.files.permission=0640","-E","logging.files.interval=1h","-E","path.data=/root/elastic-agent/data/elastic-agent-703d58/run/default/fleet-server--7.14.1"],"@timestamp":"2021-09-04T08:56:38.262Z","message":"boot"}
{"log.level":"info","service.name":"fleet-server","@timestamp":"2021-09-04T08:56:38.262Z","message":"starting communication connection back to Elastic Agent"}
{"log.level":"info","service.name":"fleet-server","@timestamp":"2021-09-04T08:56:38.262Z","message":"waiting for Elastic Agent to send initial configuration"}
{"log.level":"error","service.name":"fleet-server","error.message":"1 error: file is not a certificate adding /etc/elasticsearch/other/elasticsearch-ca.crt to the list of known CAs accessing 'output.elasticsearch'","@timestamp":"2021-09-04T08:56:38.768Z","message":"Exiting"}

Michal_Pristas · September 6, 2021, 11:27am

error  indicates a PEM file to be loaded not being a valid PEM file or certificate.

can you double check on that?

Vergil · September 21, 2021, 1:51pm

@Michal_Pristas, I am also getting "missing enrollment api key" when trying to install fleet server 7.14.1 on the same node where elasticsearch and kibana are installed.

D:\elastic-agent-7.14.1-windows-x86_64>.\elastic-agent.exe install -f --fleet-server-es=https://localhost:9200 --fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE2MzIyMjgzMDkyODc6eXQyTWk5cUVSMlM1dVlhaXRFaVJpUQ --fleet-server-policy=6bcf1aa0-0fa8-11ec-bd0d-ad88f2617109 --fleet-server-es-ca=D:\elasticsearch-7.14.1\elastic-ca.p12
2021-09-21T19:23:59.970+0545    INFO    cmd/enroll_cmd.go:336   Generating self-signed certificate for Fleet Server
2021-09-21T19:24:02.327+0545    INFO    cmd/enroll_cmd.go:650   Waiting for Elastic Agent to start Fleet Server
2021-09-21T19:24:04.379+0545    INFO    cmd/enroll_cmd.go:633   Waiting for Elastic Agent to start
2021-09-21T19:24:06.407+0545    INFO    cmd/enroll_cmd.go:683   Fleet Server - Starting
2021-09-21T19:24:08.458+0545    INFO    cmd/enroll_cmd.go:683   Fleet Server - Restarting
2021-09-21T19:24:09.473+0545    INFO    cmd/enroll_cmd.go:683   Fleet Server - Starting
2021-09-21T19:24:15.636+0545    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-21T19:24:21.717+0545    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-21T19:24:27.800+0545    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-21T19:24:33.905+0545    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-21T19:24:39.976+0545    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-21T19:24:46.047+0545    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-21T19:24:52.127+0545    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-21T19:24:58.256+0545    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-21T19:25:04.363+0545    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-21T19:25:10.583+0545    INFO    cmd/enroll_cmd.go:688   Fleet Server - Starting
2021-09-21T19:25:11.595+0545    INFO    cmd/enroll_cmd.go:664   Fleet Server - Missed last check-in
2021-09-21T19:25:11.655+0545    INFO    cmd/enroll_cmd.go:396   Starting enrollment to URL: https://WIN10E:8220/
Error: fail to enroll: fail to execute request to fleet-server: 1 error occurred:
        * missing enrollment api key

Please look into this matter.

zx8086 · September 21, 2021, 2:19pm

Have you tried removing the agent completely, installing the elastic agent and then enrolling the policy with the Fleet server ?

This worked for me after Endpoint Security screwed up my installation

Vergil · September 21, 2021, 2:48pm

I was following their guide on Add a Fleet Server.

Do you mean to start again with creating a default Fleet Server policy?

zx8086 · September 21, 2021, 3:03pm

If it is not an issue, it is an easier way to say wasting time troubleshooting a problem that might be a needle in a haystack.

Vergil · September 22, 2021, 6:04am

Did everything again. Just ran into different errors.
Bottom line - The --insecure flag does not work in latest versions.

bevano · September 23, 2021, 5:47am

Hey Vergil,

So I re-issued all my certificates (I had to since they were expiring in 6 months) and it worked. I didn't need to use the insecure flag. What I also had to add was a "--fleet-server-service-token". This is the specific command I ran to enroll it - I have blocked our all the confidential parts.

/root/elastic-agent/elastic-agent enroll --url=https://192.168.0.2:8220 \
  -f \
 --fleet-server-es=https://192.168.0.2:9200 \
 --fleet-server-service-token=THIS_WAS_90_CHARACTERS_OF_CAPITALS_LOWERCASE_AND_NUMBERS \
 --fleet-server-policy=THISWAS-NUMBERS-AND-CHARACTERS- \
 --certificate-authorities=/root/elastic-agent/ca.crt \
 --fleet-server-es-ca=/root/elastic-agent/ca.crt \
 --fleet-server-cert=/root/elastic-agent/fleet-server.crt \
 --fleet-server-cert-key=/root/elastic-agent/fleet-server.key

After running that, I was able to start it with

systemctl start elastic-agent

You can also check the status - which I found useful by typing

elastic-agent status

My cluster also started around version 7.4, so I had Beat Management (the management thing prior to fleet-server). I had to delete various API tokens before some Fleet-Server features came up.

Vergil · September 23, 2021, 10:47am

So, you just re-issued all your certificates and it worked.

bevano · September 27, 2021, 1:22pm

Yep. Probably didn’t do my certificates correctly the first time (didn’t have much SSL knowledge)

bevano · October 1, 2021, 3:33pm

Is your fleet server CA different from your elastic CA?

Maybe try this for the fleet server config within Kibana. I also had to copy the certificate into the output part.

Section 1 e. Under “Configure Fleet settings”

https://www.elastic.co/guide/en/fleet/current/secure-connections.html

Vergil · October 1, 2021, 5:22pm

Don't worry. My issue got solved just by again creating certs for the fleet server.

system · October 29, 2021, 7:22pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Fleet Missing API key Elasticsearch fleet	1	619	June 13, 2022
Error when trying to install fleet server (self-managed) Endpoint Security fleet	3	1563	September 23, 2021
Unable to install & enroll fleet server - v8.1.2 Elasticsearch fleet	1	403	May 14, 2022
Fleet server installation issue with TLS and certificates Elastic Agent fleet	0	56	October 29, 2024
Attempting to start Fleet Server fails Beats fleet	6	2242	October 24, 2021

Fleet Server - * missing enrollment api key

Related topics