Hey All,
Recently upgraded to 7.14.1 to and am struggling to install fleet server. I am running on-prem and have my self-signed certs setup. Trying to install fleet-server on the same node where elasticsearch is installed. The IP of this server is 192.168.117.101, my Kibana instance is 192.168.117.11
From the logs, I'm guessing it a problem with a certificate and being able to access Kibana/Elasticsearch to get an enrollment key? Not too sure where I can start looking since I have followed all the documentation: Encrypt traffic in a self-managed cluster | Fleet User Guide [7.14] | Elastic
I am installing it via this command
sudo /root/elastic-agent/elastic-agent enroll -f --insecure \
--url=https://192.168.117.101:8220 \
--fleet-server-es=https://192.168.117.101:9200 \
--fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE2MzA3NDA3NTM5ODQ6OTFkRm1FdXBRbXlwLWk4R3Nyd1p0UQ \
--fleet-server-policy=0d8b9ba0-0ce5-11ec-8978-2bc18b6d481d \
--certificate-authorities=/etc/elasticsearch/other/ca/ca.crt \
--fleet-server-es-ca=/etc/elasticsearch/other/elasticsearch-ca.crt \
--fleet-server-cert=/etc/elasticsearch/other/fleet-server/fleet-server.crt \
--fleet-server-cert-key=/etc/elasticsearch/other/fleet-server/fleet-server.key
The output from the console is
2021-09-04T18:56:37.214+1000 INFO cmd/enroll_cmd.go:508 Spawning Elastic Agent daemon as a subprocess to complete bootstrap process.
2021-09-04T18:56:37.337+1000 INFO application/application.go:66 Detecting execution mode
2021-09-04T18:56:37.337+1000 INFO application/application.go:87 Agent is in Fleet Server bootstrap mode
2021-09-04T18:56:37.581+1000 INFO [api] api/server.go:62 Starting stats endpoint
2021-09-04T18:56:37.581+1000 INFO application/fleet_server_bootstrap.go:124 Agent is starting
2021-09-04T18:56:37.581+1000 INFO [api] api/server.go:64 Metrics endpoint listening on: /root/elastic-agent/data/tmp/elastic-agent.sock (configured: unix:///root/elastic-agent/data/tmp/elastic-agent.sock)
2021-09-04T18:56:37.581+1000 INFO application/fleet_server_bootstrap.go:134 Agent is stopped
2021-09-04T18:56:37.582+1000 INFO stateresolver/stateresolver.go:48 New State ID is OUJQIKiU
2021-09-04T18:56:37.582+1000 INFO stateresolver/stateresolver.go:49 Converging state requires execution of 1 step(s)
2021-09-04T18:56:37.603+1000 INFO operation/operator.go:260 operation 'operation-install' skipped for fleet-server.7.14.1
2021-09-04T18:56:37.738+1000 INFO log/reporter.go:40 2021-09-04T18:56:37+10:00 - message: Application: fleet-server--7.14.1[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-09-04T18:56:37.739+1000 INFO stateresolver/stateresolver.go:66 Updating internal state
2021-09-04T18:56:38.215+1000 INFO cmd/enroll_cmd.go:683 Fleet Server - Starting
2021-09-04T18:56:38.254+1000 INFO log/reporter.go:40 2021-09-04T18:56:38+10:00 - message: Application: fleet-server--7.14.1[]: State changed to RESTARTING: exited with code: 1 - type: 'STATE' - sub_type: 'STARTING'
2021-09-04T18:56:38.254+1000 INFO log/reporter.go:40 2021-09-04T18:56:38+10:00 - message: Application: fleet-server--7.14.1[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-09-04T18:56:38.254+1000 INFO log/reporter.go:40 2021-09-04T18:56:38+10:00 - message: Application: fleet-server--7.14.1[]: State changed to RESTARTING: Restarting - type: 'STATE' - sub_type: 'STARTING'
2021-09-04T18:56:38.767+1000 INFO log/reporter.go:40 2021-09-04T18:56:38+10:00 - message: Application: fleet-server--7.14.1[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-09-04T18:56:44.219+1000 INFO cmd/enroll_cmd.go:688 Fleet Server - Starting
2021-09-04T18:56:50.224+1000 INFO cmd/enroll_cmd.go:688 Fleet Server - Starting
2021-09-04T18:56:56.229+1000 INFO cmd/enroll_cmd.go:688 Fleet Server - Starting
2021-09-04T18:57:02.234+1000 INFO cmd/enroll_cmd.go:688 Fleet Server - Starting
2021-09-04T18:57:08.238+1000 INFO cmd/enroll_cmd.go:688 Fleet Server - Starting
2021-09-04T18:57:14.242+1000 INFO cmd/enroll_cmd.go:688 Fleet Server - Starting
2021-09-04T18:57:20.248+1000 INFO cmd/enroll_cmd.go:688 Fleet Server - Starting
2021-09-04T18:57:26.253+1000 INFO cmd/enroll_cmd.go:688 Fleet Server - Starting
2021-09-04T18:57:32.258+1000 INFO cmd/enroll_cmd.go:688 Fleet Server - Starting
2021-09-04T18:57:38.272+1000 INFO cmd/enroll_cmd.go:688 Fleet Server - Starting
2021-09-04T18:57:42.589+1000 WARN status/reporter.go:236 Elastic Agent status changed to: 'degraded'
2021-09-04T18:57:42.589+1000 INFO log/reporter.go:40 2021-09-04T18:57:42+10:00 - message: Application: fleet-server--7.14.1[]: State changed to DEGRADED: Missed last check-in - type: 'STATE' - sub_type: 'RUNNING'
2021-09-04T18:57:43.276+1000 INFO cmd/enroll_cmd.go:664 Fleet Server - Missed last check-in
2021-09-04T18:57:43.276+1000 WARN [tls] tlscommon/tls_config.go:98 SSL/TLS verifications disabled.
2021-09-04T18:57:43.838+1000 INFO cmd/enroll_cmd.go:396 Starting enrollment to URL: https://192.168.117.101:8220/
2021-09-04T18:57:43.939+1000 INFO cmd/run.go:189 Shutting down Elastic Agent and sending last events...
2021-09-04T18:57:43.939+1000 INFO operation/operator.go:192 waiting for installer of pipeline 'default' to finish
2021-09-04T18:57:43.939+1000 INFO process/app.go:176 Signaling application to stop because of shutdown: fleet-server--7.14.1
2021-09-04T18:58:14.978+1000 INFO status/reporter.go:236 Elastic Agent status changed to: 'online'
2021-09-04T18:58:14.978+1000 INFO cmd/run.go:197 Shutting down completed.
2021-09-04T18:58:14.978+1000 INFO log/reporter.go:40 2021-09-04T18:58:14+10:00 - message: Application: fleet-server--7.14.1[]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'
2021-09-04T18:58:14.978+1000 INFO [api] api/server.go:66 Stats endpoint (/root/elastic-agent/data/tmp/elastic-agent.sock) finished: accept unix /root/elastic-agent/data/tmp/elastic-agent.sock: use of closed network connection
Error: fail to enroll: fail to execute request to fleet-server: 1 error occurred:
* missing enrollment api key
Fleet-Server logs
{"log.level":"info","service.name":"fleet-server","version":"7.14.1","commit":"834362b","pid":2618391,"ppid":2618352,"exe":"/root/elastic-agent/data/elastic-agent-703d58/install/fleet-server-7.14.1-linux-x86_64/fleet-server","args":["--agent-mode","-E","logging.level=info","-E","http.enabled=true","-E","http.host=unix:///root/elastic-agent/data/tmp/default/fleet-server/fleet-server.sock","-E","logging.json=true","-E","logging.ecs=true","-E","logging.files.path=/root/elastic-agent/data/elastic-agent-703d58/logs/default","-E","logging.files.name=fleet-server-json.log","-E","logging.files.keepfiles=7","-E","logging.files.permission=0640","-E","logging.files.interval=1h","-E","path.data=/root/elastic-agent/data/elastic-agent-703d58/run/default/fleet-server--7.14.1"],"@timestamp":"2021-09-04T08:56:38.262Z","message":"boot"}
{"log.level":"info","service.name":"fleet-server","@timestamp":"2021-09-04T08:56:38.262Z","message":"starting communication connection back to Elastic Agent"}
{"log.level":"info","service.name":"fleet-server","@timestamp":"2021-09-04T08:56:38.262Z","message":"waiting for Elastic Agent to send initial configuration"}
{"log.level":"error","service.name":"fleet-server","error.message":"1 error: file is not a certificate adding /etc/elasticsearch/other/elasticsearch-ca.crt to the list of known CAs accessing 'output.elasticsearch'","@timestamp":"2021-09-04T08:56:38.768Z","message":"Exiting"}