I've been trying to deploy a Fleet Server (ELK stack 7.17.6), but I'm running into a problem I cannot solve, and the logs don't give enough info for me to figure it out (also, there is nothing about this in the elasticsearch logs).
I've tried running the Fleet Server (elastic agent) both inside and outside of a container, but in both cases I get the following logs when trying to deploy:
tstrap process. 2023-05-02T15:17:44.005+0200 INFO application/application.go:67 Detecting execution mode 2023-05-02T15:17:44.006+0200 INFO application/application.go:88 Agent is in Fleet Server bootstrap mode 2023-05-02T15:17:44.223+0200 INFO [api] api/server.go:62 Starting stats endpoint 2023-05-02T15:17:44.223+0200 INFO application/fleet_server_bootstrap.go:130 Agent is starting 2023-05-02T15:17:44.224+0200 INFO [api] api/server.go:64 Metrics endpoint listening on: /var/lib/elastic-agent/data/tmp/elastic-agent.sock (configured: unix:///var/lib/elastic-agent/data/tmp/elastic-agent.sock) 2023-05-02T15:17:44.225+0200 INFO application/fleet_server_bootstrap.go:140 Agent is stopped 2023-05-02T15:17:44.233+0200 INFO stateresolver/stateresolver.go:48 New State ID is oZZvjHbN 2023-05-02T15:17:44.233+0200 INFO stateresolver/stateresolver.go:49 Converging state requires execution of 1 step(s) 2023-05-02T15:17:44.269+0200 INFO operation/operator.go:284 operation 'operation-install' skipped for fleet-server.7.17.6 2023-05-02T15:17:44.414+0200 INFO log/reporter.go:40 2023-05-02T15:17:44+02:00 - message: Application: fleet-server--7.17.6: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING' 2023-05-02T15:17:44.415+0200 INFO stateresolver/stateresolver.go:66 Updating internal state 2023-05-02T15:17:44.833+0200 INFO cmd/enroll_cmd.go:776 Fleet Server - Starting 2023-05-02T15:17:45.503+0200 ERROR status/reporter.go:236 Elastic Agent status changed to: 'error' 2023-05-02T15:17:45.503+0200 ERROR log/reporter.go:36 2023-05-02T15:17:45+02:00 - message: Application: fleet-server--7.17.6: State changed to FAILED: Error - Forbidden - type: 'ERROR' - sub_type: 'FAILED' 2023-05-02T15:17:46.837+0200 INFO cmd/enroll_cmd.go:776 Fleet Server - Error - Forbidden 2023-05-02T15:17:51.534+0200 INFO status/reporter.go:236 Elastic Agent status changed to: 'online' 2023-05-02T15:17:51.534+0200 INFO log/reporter.go:40 2023-05-02T15:17:51+02:00 - message: Application: fleet-server--7.17.6: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING' 2023-05-02T15:17:52.044+0200 ERROR status/reporter.go:236 Elastic Agent status changed to: 'error' 2023-05-02T15:17:52.045+0200 ERROR log/reporter.go:36 2023-05-02T15:17:52+02:00 - message: Application: fleet-server--7.17.6: State changed to FAILED: Error - Forbidden - type: 'ERROR' - sub_type: 'FAILED'
I've tried using service tokens, username/password, custom certs, the self-signed certs that come with the "quick start" option... etc.
The only interesting thing about the command I'm running is that I'm using "enroll" instead of "install", as I've seen people pointing out that that's how it should be done in .rpm installations. Just in case it's useful, the full command is here:
elastic-agent enroll --url=https://[KIBANA_AND_FLEET_HOSTNAME]:8220 \ --fleet-server-es=https://[ELASTIC_NODE_1]:9200 \ --fleet-server-service-token=AAEAAWVsYXN0a[REDACTED] \ --fleet-server-policy=[POLICY_ID] \ --certificate-authorities=[ROUTE_TO_CA] \ --fleet-server-es-ca=[ROUTE_TO_CA] \ --fleet-server-cert=[FLEET_SERVER_CERT] \ --fleet-server-cert-key=[FLEET_SERVER_KEY]
Even just a hint to point me into the right direction or a way to get more information about the error would be appreciated. You are also more than welcome to tell me if I'm missing any important info.