Unable to install fleet server

Dovan · August 26, 2021, 9:38pm

I'm trying to install fleet server on my local server (the same where elasticsearch is installed) and I'm not succeeding. I've tried to uninstall and reinstall elastic-agent 7.14 and it doesn't work.
the commands and the output are below:

root@abc:/home/# sudo elastic-agent enroll  -f --url=http://10.81.xxx.xxx:8220 --fleet-server-es=https://10.81.xxx.xxx:9200 --fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VxxxxxxxxxxxxxxxNDU6d21JUTUtUWxTNzYtR2xyTkZEUXNIQQ --fleet-server-policy=fc2498e0-e3fd-11eb-8662-35090124ff70 --insecure --fleet-server-es-ca=ca.crt --fleet-server-cert=fleet..crt --fleet-server-cert-key=fleet.key --enrollment-token=1bb75d8b-94ca-4b7d-b268-cc7339xxxxxx
2021-08-26T18:28:01.972-0300    INFO    cmd/enroll_cmd.go:526   Spawning Elastic Agent daemon as a subprocess to complete bootstrap process.
2021-08-26T18:28:02.193-0300    INFO    application/application.go:66   Detecting execution mode
2021-08-26T18:28:02.195-0300    INFO    application/application.go:87   Agent is in Fleet Server bootstrap mode
2021-08-26T18:28:02.977-0300    INFO    cmd/enroll_cmd.go:668   Waiting for Elastic Agent to start Fleet Server
2021-08-26T18:28:03.054-0300    INFO    [api]   api/server.go:62        Starting stats endpoint
2021-08-26T18:28:03.054-0300    INFO    application/fleet_server_bootstrap.go:124       Agent is starting
2021-08-26T18:28:03.055-0300    INFO    [api]   api/server.go:64        Metrics endpoint listening on: /var/lib/elastic-agent/data/tmp/elastic-agent.sock (configured: unix:///var/lib/elastic-agent/data/tmp/elastic-agent.sock)
2021-08-26T18:28:03.056-0300    INFO    application/fleet_server_bootstrap.go:134       Agent is stopped
2021-08-26T18:28:03.061-0300    INFO    stateresolver/stateresolver.go:48       New State ID is p2iK6r5S
2021-08-26T18:28:03.061-0300    INFO    stateresolver/stateresolver.go:49       Converging state requires execution of 1 step(s)
2021-08-26T18:28:03.225-0300    INFO    operation/operator.go:259       operation 'operation-install' skipped for fleet-server.7.14.0
2021-08-26T18:28:03.712-0300    INFO    log/reporter.go:40      2021-08-26T18:28:03-03:00 - message: Application: fleet-server--7.14.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-08-26T18:28:03.714-0300    INFO    stateresolver/stateresolver.go:66       Updating internal state
2021-08-26T18:28:03.986-0300    INFO    cmd/enroll_cmd.go:701   Fleet Server - Starting
2021-08-26T18:28:04.298-0300    INFO    log/reporter.go:40      2021-08-26T18:28:04-03:00 - message: Application: fleet-server--7.14.0[]: State changed to RESTARTING: exited with code: 1 - type: 'STATE' - sub_type: 'STARTING'
2021-08-26T18:28:04.299-0300    INFO    log/reporter.go:40      2021-08-26T18:28:04-03:00 - message: Application: fleet-server--7.14.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-08-26T18:28:04.299-0300    INFO    log/reporter.go:40      2021-08-26T18:28:04-03:00 - message: Application: fleet-server--7.14.0[]: State changed to RESTARTING: Restarting - type: 'STATE' - sub_type: 'STARTING'
2021-08-26T18:28:04.875-0300    INFO    log/reporter.go:40      2021-08-26T18:28:04-03:00 - message: Application: fleet-server--7.14.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-08-26T18:28:10.001-0300    INFO    cmd/enroll_cmd.go:706   Fleet Server - Starting
2021-08-26T18:28:16.016-0300    INFO    cmd/enroll_cmd.go:706   Fleet Server - Starting
2021-08-26T18:28:22.035-0300    INFO    cmd/enroll_cmd.go:706   Fleet Server - Starting
2021-08-26T18:28:28.056-0300    INFO    cmd/enroll_cmd.go:706   Fleet Server - Starting
2021-08-26T18:28:34.070-0300    INFO    cmd/enroll_cmd.go:706   Fleet Server - Starting
2021-08-26T18:28:40.083-0300    INFO    cmd/enroll_cmd.go:706   Fleet Server - Starting
2021-08-26T18:28:46.098-0300    INFO    cmd/enroll_cmd.go:706   Fleet Server - Starting
2021-08-26T18:28:52.111-0300    INFO    cmd/enroll_cmd.go:706   Fleet Server - Starting
2021-08-26T18:28:58.124-0300    INFO    cmd/enroll_cmd.go:706   Fleet Server - Starting
2021-08-26T18:29:04.144-0300    INFO    cmd/enroll_cmd.go:706   Fleet Server - Starting
2021-08-26T18:29:08.067-0300    WARN    status/reporter.go:236  Elastic Agent status changed to: 'degraded'
2021-08-26T18:29:08.067-0300    INFO    log/reporter.go:40      2021-08-26T18:29:08-03:00 - message: Application: fleet-server--7.14.0[]: State changed to DEGRADED: Missed last check-in - type: 'STATE' - sub_type: 'RUNNING'
2021-08-26T18:29:08.153-0300    INFO    cmd/enroll_cmd.go:682   Fleet Server - Missed last check-in
2021-08-26T18:29:08.153-0300    WARN    [tls]   tlscommon/tls_config.go:98      SSL/TLS verifications disabled.
2021-08-26T18:29:08.344-0300    INFO    cmd/enroll_cmd.go:414   Starting enrollment to URL: http://10.81.xxx.xxx:8220/
2021-08-26T18:29:08.448-0300    WARN    cmd/enroll_cmd.go:425   Remote server is not ready to accept connections, will retry in a moment.
2021-08-26T18:30:08.075-0300    ERROR   status/reporter.go:236  Elastic Agent status changed to: 'error'
2021-08-26T18:30:08.075-0300    ERROR   log/reporter.go:36      2021-08-26T18:30:08-03:00 - message: Application: fleet-server--7.14.0[]: State changed to FAILED: Missed two check-ins - type: 'ERROR' - sub_type: 'FAILED'
2021-08-26T18:30:08.449-0300    INFO    cmd/enroll_cmd.go:432   Retrying enrollment to URL: http://10.81.200.160:8220/
2021-08-26T18:30:08.554-0300    WARN    cmd/enroll_cmd.go:425   Remote server is not ready to accept connections, will retry in a moment.
2021-08-26T18:32:08.556-0300    INFO    cmd/enroll_cmd.go:432   Retrying enrollment to URL: http://10.81.200.160:8220/
2021-08-26T18:32:08.659-0300    WARN    cmd/enroll_cmd.go:425   Remote server is not ready to accept connections, will retry in a moment.

and gets stuck on the "Remote server is not ready to accept connections, will retry in a moment". forever

Dovan · August 26, 2021, 9:49pm

on a second run without some flags I'm having this error: "missing enrollment api key"

root@abc:/home/# sudo elastic-agent enroll  -f --url=https://10.81.xxx.xxx:8220 --fleet-server-es=https://10.81.xxx.xxx:9200 --fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VxxxxxxxxxxxxxxxNDU6d21JUTUtUWxTNzYtR2xyTkZEUXNIQQ --fleet-server-policy=fc2498e0-e3fd-11eb-8662-35090124ff70 --insecure --fleet-server-es-ca=ca.crt
2021-08-26T18:41:09.090-0300    INFO    cmd/enroll_cmd.go:354   Generating self-signed certificate for Fleet Server
2021-08-26T18:41:09.954-0300    INFO    cmd/enroll_cmd.go:526   Spawning Elastic Agent daemon as a subprocess to complete bootstrap process.
2021-08-26T18:41:10.139-0300    INFO    application/application.go:66   Detecting execution mode
2021-08-26T18:41:10.141-0300    INFO    application/application.go:87   Agent is in Fleet Server bootstrap mode
2021-08-26T18:41:10.958-0300    INFO    cmd/enroll_cmd.go:668   Waiting for Elastic Agent to start Fleet Server
2021-08-26T18:41:11.020-0300    INFO    [api]   api/server.go:62        Starting stats endpoint
2021-08-26T18:41:11.020-0300    INFO    application/fleet_server_bootstrap.go:124       Agent is starting
2021-08-26T18:41:11.020-0300    INFO    [api]   api/server.go:64        Metrics endpoint listening on: /var/lib/elastic-agent/data/tmp/elastic-agent.sock (configured: unix:///var/lib/elastic-agent/data/tmp/elastic-agent.sock)
2021-08-26T18:41:11.021-0300    INFO    application/fleet_server_bootstrap.go:134       Agent is stopped
2021-08-26T18:41:11.025-0300    INFO    stateresolver/stateresolver.go:48       New State ID is Etq-nzJP
2021-08-26T18:41:11.025-0300    INFO    stateresolver/stateresolver.go:49       Converging state requires execution of 1 step(s)
2021-08-26T18:41:11.132-0300    INFO    operation/operator.go:259       operation 'operation-install' skipped for fleet-server.7.14.0
2021-08-26T18:41:12.107-0300    INFO    log/reporter.go:40      2021-08-26T18:41:12-03:00 - message: Application: fleet-server--7.14.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-08-26T18:41:12.110-0300    INFO    stateresolver/stateresolver.go:66       Updating internal state
2021-08-26T18:41:12.693-0300    INFO    log/reporter.go:40      2021-08-26T18:41:12-03:00 - message: Application: fleet-server--7.14.0[]: State changed to RESTARTING: exited with code: 1 - type: 'STATE' - sub_type: 'STARTING'
2021-08-26T18:41:12.694-0300    INFO    log/reporter.go:40      2021-08-26T18:41:12-03:00 - message: Application: fleet-server--7.14.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-08-26T18:41:12.694-0300    INFO    log/reporter.go:40      2021-08-26T18:41:12-03:00 - message: Application: fleet-server--7.14.0[]: State changed to RESTARTING: Restarting - type: 'STATE' - sub_type: 'STARTING'
2021-08-26T18:41:12.964-0300    INFO    cmd/enroll_cmd.go:701   Fleet Server - Restarting
2021-08-26T18:41:13.260-0300    INFO    log/reporter.go:40      2021-08-26T18:41:13-03:00 - message: Application: fleet-server--7.14.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-08-26T18:41:13.965-0300    INFO    cmd/enroll_cmd.go:701   Fleet Server - Starting
2021-08-26T18:41:19.978-0300    INFO    cmd/enroll_cmd.go:706   Fleet Server - Starting
2021-08-26T18:41:25.992-0300    INFO    cmd/enroll_cmd.go:706   Fleet Server - Starting
2021-08-26T18:41:32.004-0300    INFO    cmd/enroll_cmd.go:706   Fleet Server - Starting
2021-08-26T18:41:38.016-0300    INFO    cmd/enroll_cmd.go:706   Fleet Server - Starting
2021-08-26T18:41:44.028-0300    INFO    cmd/enroll_cmd.go:706   Fleet Server - Starting
2021-08-26T18:41:50.040-0300    INFO    cmd/enroll_cmd.go:706   Fleet Server - Starting
2021-08-26T18:41:56.055-0300    INFO    cmd/enroll_cmd.go:706   Fleet Server - Starting
2021-08-26T18:42:02.070-0300    INFO    cmd/enroll_cmd.go:706   Fleet Server - Starting
2021-08-26T18:42:08.084-0300    INFO    cmd/enroll_cmd.go:706   Fleet Server - Starting
2021-08-26T18:42:14.102-0300    INFO    cmd/enroll_cmd.go:706   Fleet Server - Starting
2021-08-26T18:42:16.030-0300    INFO    log/reporter.go:40      2021-08-26T18:42:16-03:00 - message: Application: fleet-server--7.14.0[]: State changed to DEGRADED: Missed last check-in - type: 'STATE' - sub_type: 'RUNNING'
2021-08-26T18:42:16.030-0300    WARN    status/reporter.go:236  Elastic Agent status changed to: 'degraded'
2021-08-26T18:42:16.106-0300    INFO    cmd/enroll_cmd.go:682   Fleet Server - Missed last check-in
2021-08-26T18:42:16.107-0300    WARN    [tls]   tlscommon/tls_config.go:98      SSL/TLS verifications disabled.
2021-08-26T18:42:16.926-0300    INFO    cmd/enroll_cmd.go:414   Starting enrollment to URL: https://abc:8220/
2021-08-26T18:42:17.029-0300    INFO    cmd/run.go:189  Shutting down Elastic Agent and sending last events...
2021-08-26T18:42:17.029-0300    INFO    operation/operator.go:191       waiting for installer of pipeline 'default' to finish
2021-08-26T18:42:17.029-0300    INFO    process/app.go:181      Signaling application to stop because of shutdown: fleet-server--7.14.0
2021-08-26T18:42:47.076-0300    INFO    status/reporter.go:236  Elastic Agent status changed to: 'online'
2021-08-26T18:42:47.076-0300    INFO    log/reporter.go:40      2021-08-26T18:42:47-03:00 - message: Application: fleet-server--7.14.0[]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'
2021-08-26T18:42:47.076-0300    INFO    cmd/run.go:197  Shutting down completed.
2021-08-26T18:42:47.077-0300    INFO    [api]   api/server.go:66        Stats endpoint (/var/lib/elastic-agent/data/tmp/elastic-agent.sock) finished: accept unix /var/lib/elastic-agent/data/tmp/elastic-agent.sock: use of closed network connection
Error: fail to enroll: fail to execute request to fleet-server: 1 error occurred:
        * missing enrollment api key

TheHunter1 · August 31, 2021, 2:23pm

I have the same problem,
I am trying to install the fleet server, and keep getting remote server is not ready to accept connections, will retry in a moment

I downloaded the debian version of elastic agent, then Installed it

dpkg -i elastic-agent-7.14.0-amd64.deb

Then run the command :

 sudo elastic-agent enroll --url=https://X.X.X.X:8220 -f --fleet-server-es=https://Y.Y.Y.Y:9200 --fleet-server-service-token=************** --fleet-server-policy=******* --certificate-authorities=ca.crt --fleet-server-es-ca=elastic.crt --fleet-server-cert=fleet.crt --fleet-server-cert-key=fleet.key.key --enrollment-token=*****

Did find how to solve the problem ?

Best regards

blaker · September 1, 2021, 12:44pm

I believe the issue is that you're not using absolute paths to the certificates and CA's. That is a known issue that Elastic Agent needs to solve, but if you switch to using absolute paths it should resolve your issue for now.

github.com/elastic/beats

Fleet Server install can fail when using relative paths for certificates

opened 06:15PM - 25 Aug 21 UTC

n0othing

bug Team:Agent

- Version: `7.14.0` - Operating System: MacOS 11.5.2 - Steps to Reproduce: … Attempting to install Fleet Server using relative certificate file paths results in the install failing, with no clear logging as to why: ```bash sudo ./elastic-agent install --url=https://127.0.0.1:8220 \ -f \ --fleet-server-es=https://127.0.0.1:9200 \ --fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE2MjkyMjE2MjU1NzU6UG81UVp6MFFTVTZFa1JtYk4tbWYxUQ \ --fleet-server-policy=2ab0ceb0-ff7c-11eb-8a64-5f3c299c93d0 \ --certificate-authorities=certs/ca.crt \ --fleet-server-es-ca=certs/ca.crt \ --fleet-server-cert=certs/fleet-server.crt \ --fleet-server-cert-key=certs/fleet-server.key ``` ``` 2021-08-25T13:13:14.989-0400 INFO cmd/enroll_cmd.go:651 Waiting for Elastic Agent to start 2021-08-25T13:13:15.994-0400 INFO cmd/enroll_cmd.go:701 Fleet Server - Starting 2021-08-25T13:13:16.995-0400 INFO cmd/enroll_cmd.go:701 Fleet Server - Restarting 2021-08-25T13:13:17.997-0400 INFO cmd/enroll_cmd.go:701 Fleet Server - Starting 2021-08-25T13:13:24.017-0400 INFO cmd/enroll_cmd.go:706 Fleet Server - Starting 2021-08-25T13:13:30.031-0400 INFO cmd/enroll_cmd.go:706 Fleet Server - Starting 2021-08-25T13:13:36.052-0400 INFO cmd/enroll_cmd.go:706 Fleet Server - Starting 2021-08-25T13:13:42.072-0400 INFO cmd/enroll_cmd.go:706 Fleet Server - Starting 2021-08-25T13:13:48.096-0400 INFO cmd/enroll_cmd.go:706 Fleet Server - Starting 2021-08-25T13:13:54.112-0400 INFO cmd/enroll_cmd.go:706 Fleet Server - Starting 2021-08-25T13:14:00.135-0400 INFO cmd/enroll_cmd.go:706 Fleet Server - Starting 2021-08-25T13:14:06.154-0400 INFO cmd/enroll_cmd.go:706 Fleet Server - Starting 2021-08-25T13:14:12.173-0400 INFO cmd/enroll_cmd.go:706 Fleet Server - Starting 2021-08-25T13:14:18.190-0400 INFO cmd/enroll_cmd.go:706 Fleet Server - Starting 2021-08-25T13:14:21.197-0400 INFO cmd/enroll_cmd.go:682 Fleet Server - Missed last check-in 2021-08-25T13:14:21.520-0400 INFO cmd/enroll_cmd.go:414 Starting enrollment to URL: https://127.0.0.1:8220/ Error: fail to enroll: fail to execute request to fleet-server: 1 error occurred: * missing enrollment api key Error: enroll command failed with exit code: 1 ``` The `/Library/Elastic/Agent` directory gets removed after this failure so we're unable to review the logs to see what might have gone wrong. By adding an `--enrollment-token` to the install command, the install still fails, but the agent stays up allowing us to investigate the log directory: ```diff sudo ./elastic-agent install --url=https://127.0.0.1:8220 \ -f \ --fleet-server-es=https://127.0.0.1:9200 \ --fleet-server-service-token=AAEAAWVsYXN0aWMvZmxlZXQtc2VydmVyL3Rva2VuLTE2MjkyMjE2MjU1NzU6UG81UVp6MFFTVTZFa1JtYk4tbWYxUQ \ --fleet-server-policy=2ab0ceb0-ff7c-11eb-8a64-5f3c299c93d0 \ --certificate-authorities=certs/ca.crt \ --fleet-server-es-ca=certs/ca.crt \ --fleet-server-cert=certs/fleet-server.crt \ --fleet-server-cert-key=certs/fleet-server.key \ + --enrollment-token=c1lrTFZYc0I3LUR3eWpNdnVfV0o6ay1yNDdKWjNTRTZKbi1sZkw3VF9Rdw== ``` ``` 2021-08-25T14:08:25.057-0400 INFO cmd/enroll_cmd.go:668 Waiting for Elastic Agent to start Fleet Server 2021-08-25T14:08:27.064-0400 INFO cmd/enroll_cmd.go:701 Fleet Server - Starting 2021-08-25T14:08:28.066-0400 INFO cmd/enroll_cmd.go:701 Fleet Server - Restarting 2021-08-25T14:08:29.070-0400 INFO cmd/enroll_cmd.go:701 Fleet Server - Starting 2021-08-25T14:08:35.089-0400 INFO cmd/enroll_cmd.go:706 Fleet Server - Starting 2021-08-25T14:08:41.101-0400 INFO cmd/enroll_cmd.go:706 Fleet Server - Starting 2021-08-25T14:08:47.110-0400 INFO cmd/enroll_cmd.go:706 Fleet Server - Starting 2021-08-25T14:08:53.126-0400 INFO cmd/enroll_cmd.go:706 Fleet Server - Starting 2021-08-25T14:08:59.141-0400 INFO cmd/enroll_cmd.go:706 Fleet Server - Starting 2021-08-25T14:09:05.167-0400 INFO cmd/enroll_cmd.go:706 Fleet Server - Starting 2021-08-25T14:09:11.186-0400 INFO cmd/enroll_cmd.go:706 Fleet Server - Starting 2021-08-25T14:09:17.199-0400 INFO cmd/enroll_cmd.go:706 Fleet Server - Starting 2021-08-25T14:09:23.218-0400 INFO cmd/enroll_cmd.go:706 Fleet Server - Starting 2021-08-25T14:09:29.237-0400 INFO cmd/enroll_cmd.go:706 Fleet Server - Starting 2021-08-25T14:09:31.242-0400 INFO cmd/enroll_cmd.go:682 Fleet Server - Missed last check-in 2021-08-25T14:09:31.354-0400 INFO cmd/enroll_cmd.go:414 Starting enrollment to URL: https://127.0.0.1:8220/ 2021-08-25T14:09:31.461-0400 WARN cmd/enroll_cmd.go:425 Remote server is not ready to accept connections, will retry in a moment. ``` ``` cat /Library/Elastic/Agent/data/elastic-agent-e127fc/logs/default/fleet-server-json.log ``` ```json {"log.level":"info","service.name":"fleet-server","version":"7.14.0","commit":"82d6804","pid":17409,"ppid":17403,"exe":"/Library/Elastic/Agent/data/elastic-agent-e127fc/install/fleet-server-7.14.0-darwin-x86_64/fleet-server","args":["--agent-mode","-E","logging.level=info","-E","http.enabled=true","-E","http.host=unix:///Library/Elastic/Agent/data/tmp/default/fleet-server/fleet-server.sock","-E","logging.json=true","-E","logging.ecs=true","-E","logging.files.path=/Library/Elastic/Agent/data/elastic-agent-e127fc/logs/default","-E","logging.files.name=fleet-server-json.log","-E","logging.files.keepfiles=7","-E","logging.files.permission=0640","-E","logging.files.interval=1h","-E","path.data=/Library/Elastic/Agent/data/elastic-agent-e127fc/run/default/fleet-server--7.14.0"],"@timestamp":"2021-08-25T18:08:27.659Z","message":"boot"} {"log.level":"info","service.name":"fleet-server","@timestamp":"2021-08-25T18:08:27.661Z","message":"starting communication connection back to Elastic Agent"} {"log.level":"info","service.name":"fleet-server","@timestamp":"2021-08-25T18:08:27.661Z","message":"waiting for Elastic Agent to send initial configuration"} {"log.level":"error","service.name":"fleet-server","error.message":"1 error: open certs/ca.crt: no such file or directory reading <nil> accessing 'output.elasticsearch'","@timestamp":"2021-08-25T18:08:28.245Z","message":"Exiting"} ```

TheHunter1 · September 1, 2021, 1:16pm

Thanks for your answer @blaker ,

Yes I confirm, as when I used the absolute paths to the certificates and CA's the error changed to X509 certificate signed by unknown authority
I will dig more into this issue

Thanks again ^ ^

blaker · September 1, 2021, 1:29pm

@TheHunter1 You need to add the --certificate-authorities paramater and give it the absolute path to the CA. The Elastic Agent needs the CA to be able to verify the certificate that you are giving the Fleet Server to start with.

TheHunter1 · September 1, 2021, 1:50pm

@blaker , for me I have a cluster with 5 nodes: should I mention all the nodes in the --fleet-server-es parameter or just one of them ?

for me right now I am declaring just one master node , so my certificates are :

ca.crt : certificate authorities
Master-01.crt : Elastic node certificate
fleet-server.crt :
fleet-server.key

So to run the fleet server, I run this command :

sudo elastic-agent enroll --url=https://X.X.X.X:8220 -f --fleet-server-es=https://Y.Y.Y.Y:9200 --fleet-server-service-token=*********** --fleet-server-policy=******** --certificate-authorities=/home/hunter/elasticagent/ca.crt --fleet-server-es-ca=/home/hunter/elasticagent/MASTER-01.crt --fleet-server-cert=/home/hunter/elasticagent/fleet-server/fleet-server.crt --fleet-server-cert-key=/home/hunter/elasticagent/fleet-server/fleet-server.key --enrollment-token=********

But i still getting errors :

2021-09-01T15:19:07.778+0200	ERROR	status/reporter.go:236	Elastic Agent status changed to: 'error'
2021-09-01T15:19:07.778+0200	ERROR	log/reporter.go:36	2021-09-01T15:19:07+02:00 - message: Application: fleet-server--7.14.0[]: State changed to FAILED: Error - x509: certificate signed by unknown authority - type: 'ERROR' - sub_type: 'FAILED'
2021-09-01T15:19:08.434+0200	INFO	cmd/enroll_cmd.go:701	Fleet Server - Error - x509: certificate signed by unknown authority
2021-09-01T15:19:14.438+0200	INFO	cmd/enroll_cmd.go:706	Fleet Server - Error - x509: certificate signed by unknown authority

Even when I tried to run with --insecure option still have the same issue:

sudo elastic-agent enroll -f --fleet-server-es=https://X.X.X.X:9200 --fleet-server-service-token=******** --fleet-server-policy=******* --insecure

Knowing that I am in a debian machine, so already installed the elasticagent with the command :

dpkg -i elastic-agent-7.14.0-amd64.deb

and I have created my fleet-server certificate with the command :

bin/elasticsearch-certutil cert --name fleet-server --ca-cert config/certs/ca/ca.crt --ca-key config/certs/ca/ca.key --dns MachineName --ip X.X.X.X --pem

blaker · September 1, 2021, 2:28pm

Fleet Server - Error - x509: certificate signed by unknown authority shows that the issue is with Fleet Server communicating with elasticsearch. The IP address you are giving in --fleet-server-es needs to match the DNS name that is given to the certificate that you used to get elasticsearch up and running. If you cannot do that using --insecure can help you bypass that issue for now just to see it all working. Obviously you don't want that in a production installation.

TheHunter1 · September 1, 2021, 2:57pm

So to try that, I run it in my dev machine which is running Elasticsearch from source (version 8) and I am getting new errors which means that I resolved the X509 certificate errors

The new errors are :

2021-09-01T16:53:31.440+0200	ERROR	status/reporter.go:236	Elastic Agent status changed to: 'error'
2021-09-01T16:53:31.440+0200	ERROR	log/reporter.go:36	2021-09-01T16:53:31+02:00 - message: Application: fleet-server--7.14.0[]: State changed to FAILED: Error - elastic fail 403:security_exception:action [indices:monitor/fleet/global_checkpoints] is unauthorized for user [elastic/fleet-server] on indices [.fleet-actions], this action is granted by the index privileges [monitor,manage,all] - type: 'ERROR' - sub_type: 'FAILED'
2021-09-01T16:53:32.137+0200	INFO	cmd/enroll_cmd.go:701	Fleet Server - Error - elastic fail 403:security_exception:action [indices:monitor/fleet/global_checkpoints] is unauthorized for user [elastic/fleet-server] on indices [.fleet-actions], this action is granted by the index privileges [monitor,manage,all]
2021-09-01T16:53:38.142+0200	INFO	cmd/enroll_cmd.go:706	Fleet Server - Error - elastic fail 403:security_exception:action [indices:monitor/fleet/global_checkpoints] is unauthorized for user [elastic/fleet-server] on indices [.fleet-actions], this action is granted by the index privileges [monitor,manage,all]
2021-09-01T16:53:41.440+0200	INFO	status/reporter.go:236	Elastic Agent status changed to: 'online'
2021-09-01T16:53:41.441+0200	INFO	log/reporter.go:40	2021-09-01T16:53:41+02:00 - message: Application: fleet-server--7.14.0[]: State changed to RESTARTING:  - type: 'STATE' - sub_type: 'STARTING'
2021-09-01T16:53:41.441+0200	INFO	log/reporter.go:40	2021-09-01T16:53:41+02:00 - message: Application: fleet-server--7.14.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-09-01T16:53:41.441+0200	INFO	log/reporter.go:40	2021-09-01T16:53:41+02:00 - message: Application: fleet-server--7.14.0[]: State changed to RESTARTING: Restarting - type: 'STATE' - sub_type: 'STARTING'
2021-09-01T16:53:41.959+0200	INFO	log/reporter.go:40	2021-09-01T16:53:41+02:00 - message: Application: fleet-server--7.14.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-09-01T16:53:42.145+0200	INFO	cmd/enroll_cmd.go:701	Fleet Server - Starting
2021-09-01T16:53:42.460+0200	ERROR	status/reporter.go:236	Elastic Agent status changed to: 'error'
2021-09-01T16:53:42.460+0200	ERROR	log/reporter.go:36	2021-09-01T16:53:42+02:00 - message: Application: fleet-server--7.14.0[]: State changed to FAILED: Error - elastic fail 403:security_exception:action [indices:monitor/fleet/global_checkpoints] is unauthorized for user [elastic/fleet-server] on indices [.fleet-actions], this action is granted by the index privileges [monitor,manage,all] - type: 'ERROR' - sub_type: 'FAILED'
2021-09-01T16:53:43.146+0200	INFO	cmd/enroll_cmd.go:701	Fleet Server - Error - elastic fail 403:security_exception:action [indices:monitor/fleet/global_checkpoints] is unauthorized for user [elastic/fleet-server] on indices [.fleet-actions], this action is granted by the index privileges [monitor,manage,all]
2021-09-01T16:53:49.151+0200	INFO	cmd/enroll_cmd.go:706	Fleet Server - Error - elastic fail 403:security_exception:action [indices:monitor/fleet/global_checkpoints] is unauthorized for user [elastic/fleet-server] on indices [.fleet-actions], this action is granted by the index privileges [monitor,manage,all]

Dovan · September 1, 2021, 6:57pm

hello @blaker , yes you were right! thanks, i used the path to the certificate and managed to install it, but now the fleet server goes from offline to online several times and is not stabilized.
here are the latest logs:

{"log.level":"error","@timestamp":"2021-09-01T16:49:07.296Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":205},"message":"Could not communicate with fleet-server Checking API will retry, error: status code: 400, fleet-server returned an error: BadRequest","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-09-01T16:56:14.787Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":205},"message":"Could not communicate with fleet-server Checking API will retry, error: status code: 400, fleet-server returned an error: BadRequest","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-09-01T17:02:26.531Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":205},"message":"Could not communicate with fleet-server Checking API will retry, error: status code: 400, fleet-server returned an error: BadRequest","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-09-01T17:08:47.116Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":205},"message":"Could not communicate with fleet-server Checking API will retry, error: status code: 400, fleet-server returned an error: BadRequest","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-09-01T17:18:10.367Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":205},"message":"Could not communicate with fleet-server Checking API will retry, error: status code: 400, fleet-server returned an error: BadRequest","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-09-01T17:27:25.258Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":205},"message":"Could not communicate with fleet-server Checking API will retry, error: status code: 400, fleet-server returned an error: BadRequest","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-09-01T17:34:12.979Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":205},"message":"Could not communicate with fleet-server Checking API will retry, error: status code: 400, fleet-server returned an error: BadRequest","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-09-01T17:44:00.301Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":205},"message":"Could not communicate with fleet-server Checking API will retry, error: status code: 400, fleet-server returned an error: BadRequest","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-09-01T17:53:43.708Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":205},"message":"Could not communicate with fleet-server Checking API will retry, error: status code: 400, fleet-server returned an error: BadRequest","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-09-01T18:01:33.919Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":205},"message":"Could not communicate with fleet-server Checking API will retry, error: status code: 400, fleet-server returned an error: BadRequest","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-09-01T18:10:30.089Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":205},"message":"Could not communicate with fleet-server Checking API will retry, error: status code: 400, fleet-server returned an error: BadRequest","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-09-01T18:19:06.122Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":205},"message":"Could not communicate with fleet-server Checking API will retry, error: status code: 400, fleet-server returned an error: BadRequest","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-09-01T18:28:37.007Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":205},"message":"Could not communicate with fleet-server Checking API will retry, error: status code: 400, fleet-server returned an error: BadRequest","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-09-01T18:36:06.076Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":205},"message":"Could not communicate with fleet-server Checking API will retry, error: status code: 400, fleet-server returned an error: BadRequest","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-09-01T18:41:15.563Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":205},"message":"Could not communicate with fleet-server Checking API will retry, error: status code: 400, fleet-server returned an error: BadRequest","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-09-01T18:49:01.328Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":205},"message":"Could not communicate with fleet-server Checking API will retry, error: status code: 400, fleet-server returned an error: BadRequest","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-09-01T18:55:10.978Z","log.origin":{"file.name":"fleet/fleet_gateway.go","file.line":205},"message":"Could not communicate with fleet-server Checking API will retry, error: status code: 400, fleet-server returned an error: BadRequest","ecs.version":"1.6.0"}

system · September 29, 2021, 6:57pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Unable to install fleet server Beats fleet	6	3946	September 23, 2021
Unable to install fleet server Beats fleet	2	717	September 23, 2021
Fleet server Installation error Beats fleet	4	1098	November 26, 2021
Error installing Fleet Server Beats elastic-agent	5	1265	June 27, 2022
Fleet Server & ES Agent Elasticsearch elastic-stack-security , fleet	5	987	December 5, 2022

Unable to install fleet server

Related topics