Error when installing/enrolling Fleet Server

Miguel_Azorin · May 2, 2023, 2:07pm

Hi,

I've been trying to deploy a Fleet Server (ELK stack 7.17.6), but I'm running into a problem I cannot solve, and the logs don't give enough info for me to figure it out (also, there is nothing about this in the elasticsearch logs).

I've tried running the Fleet Server (elastic agent) both inside and outside of a container, but in both cases I get the following logs when trying to deploy:

tstrap process.
2023-05-02T15:17:44.005+0200    INFO    application/application.go:67   Detecting execution mode
2023-05-02T15:17:44.006+0200    INFO    application/application.go:88   Agent is in Fleet Server bootstrap mode
2023-05-02T15:17:44.223+0200    INFO    [api]   api/server.go:62        Starting stats endpoint
2023-05-02T15:17:44.223+0200    INFO    application/fleet_server_bootstrap.go:130       Agent is starting
2023-05-02T15:17:44.224+0200    INFO    [api]   api/server.go:64        Metrics endpoint listening on: /var/lib/elastic-agent/data/tmp/elastic-agent.sock (configured: unix:///var/lib/elastic-agent/data/tmp/elastic-agent.sock)
2023-05-02T15:17:44.225+0200    INFO    application/fleet_server_bootstrap.go:140       Agent is stopped
2023-05-02T15:17:44.233+0200    INFO    stateresolver/stateresolver.go:48       New State ID is oZZvjHbN
2023-05-02T15:17:44.233+0200    INFO    stateresolver/stateresolver.go:49       Converging state requires execution of 1 step(s)
2023-05-02T15:17:44.269+0200    INFO    operation/operator.go:284       operation 'operation-install' skipped for fleet-server.7.17.6
2023-05-02T15:17:44.414+0200    INFO    log/reporter.go:40      2023-05-02T15:17:44+02:00 - message: Application: fleet-server--7.17.6[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2023-05-02T15:17:44.415+0200    INFO    stateresolver/stateresolver.go:66       Updating internal state
2023-05-02T15:17:44.833+0200    INFO    cmd/enroll_cmd.go:776   Fleet Server - Starting
2023-05-02T15:17:45.503+0200    ERROR   status/reporter.go:236  Elastic Agent status changed to: 'error'
2023-05-02T15:17:45.503+0200    ERROR   log/reporter.go:36      2023-05-02T15:17:45+02:00 - message: Application: fleet-server--7.17.6[]: State changed to FAILED: Error - Forbidden - type: 'ERROR' - sub_type: 'FAILED'
2023-05-02T15:17:46.837+0200    INFO    cmd/enroll_cmd.go:776   Fleet Server - Error - Forbidden
2023-05-02T15:17:51.534+0200    INFO    status/reporter.go:236  Elastic Agent status changed to: 'online'
2023-05-02T15:17:51.534+0200    INFO    log/reporter.go:40      2023-05-02T15:17:51+02:00 - message: Application: fleet-server--7.17.6[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2023-05-02T15:17:52.044+0200    ERROR   status/reporter.go:236  Elastic Agent status changed to: 'error'
2023-05-02T15:17:52.045+0200    ERROR   log/reporter.go:36      2023-05-02T15:17:52+02:00 - message: Application: fleet-server--7.17.6[]: State changed to FAILED: Error - Forbidden - type: 'ERROR' - sub_type: 'FAILED'

I've tried using service tokens, username/password, custom certs, the self-signed certs that come with the "quick start" option... etc.

The only interesting thing about the command I'm running is that I'm using "enroll" instead of "install", as I've seen people pointing out that that's how it should be done in .rpm installations. Just in case it's useful, the full command is here:

elastic-agent enroll --url=https://[KIBANA_AND_FLEET_HOSTNAME]:8220 \
  --fleet-server-es=https://[ELASTIC_NODE_1]:9200 \
  --fleet-server-service-token=AAEAAWVsYXN0a[REDACTED] \
  --fleet-server-policy=[POLICY_ID] \
  --certificate-authorities=[ROUTE_TO_CA] \
  --fleet-server-es-ca=[ROUTE_TO_CA] \
  --fleet-server-cert=[FLEET_SERVER_CERT] \
  --fleet-server-cert-key=[FLEET_SERVER_KEY]

Even just a hint to point me into the right direction or a way to get more information about the error would be appreciated. You are also more than welcome to tell me if I'm missing any important info.

Best regards,

Miguel

stephenb · May 2, 2023, 10:55pm

Hi @Miguel_Azorin

We would need to see the actual agent logs.. not sure (apologies) what tstrap is but those logs don't really look like the raw agent logs ...

I would probably get this all running with tar.gz install and then convert over to .rpm / .dev

When you install with the tar.gz I think you will see the logs in the foreground.

Just my suggestion

Miguel_Azorin · May 3, 2023, 8:45am

I apologize, I truncated the first log line accidentally, leaving only strap. The lines I truncated are:

This will replace your current settings. Do you want to continue? [Y/n]:y
2023-05-03T10:25:11.392+0200    INFO    cmd/enroll_cmd.go:571   Spawning Elastic Agent daemon as a subprocess to complete bootstrap process.

These logs are printed in the foreground when I execute elastic-agent enroll [PARAMETERS]. Aren't these the raw agent logs? If not, I will just install the tar.gz version, just as you suggested.

stephenb · May 3, 2023, 2:05pm

I would install with tar.gz

Miguel_Azorin · May 4, 2023, 9:58am

And you would be right.

It is now working, using the same parameters, with the only difference that I've installed it using the tar.gz

Thanks for the help, this would have been pretty hard to figure out for me!

system · June 1, 2023, 9:59am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.