Fleet Server keeps crashing

So I wanted to explore the new fleet changes which where introduced lately I dod a complete new setup to get hands on with it. But I am not able to get the fleet server running it keeps crashing with exit code 1 and I am not getting any more information than this in the logs too.
Mine is a on prem setup so has anyone got the fleet server running in an on prem setup yet?

2021-05-31T10:22:46.776+0530    INFO    cmd/enroll_cmd.go:300   Generating self-signed certificate for Fleet Server
2021-05-31T10:22:48.786+0530    INFO    cmd/enroll_cmd.go:468   Spawning Elastic Agent daemon as a subprocess to complete bootstrap process.
2021-05-31T10:22:49.044+0530    INFO    warn/warn.go:18 The Elastic Agent is currently in BETA and should not be used in production
2021-05-31T10:22:49.045+0530    INFO    application/application.go:68   Detecting execution mode
2021-05-31T10:22:49.047+0530    INFO    application/application.go:89   Agent is in Fleet Server bootstrap mode
2021-05-31T10:22:49.791+0530    INFO    cmd/enroll_cmd.go:610   Waiting for Elastic Agent to start Fleet Server
2021-05-31T10:22:50.175+0530    INFO    [api]   api/server.go:62        Starting stats endpoint
2021-05-31T10:22:50.175+0530    INFO    application/fleet_server_bootstrap.go:124       Agent is starting
2021-05-31T10:22:50.176+0530    INFO    [api]   api/server.go:64        Metrics endpoint listening on: /var/lib/elastic-agent/data/tmp/elastic-agent.sock (configured: unix:///var/lib/elastic-agent/data/tmp/elastic-agent.sock)
2021-05-31T10:22:50.178+0530    INFO    application/fleet_server_bootstrap.go:134       Agent is stopped
2021-05-31T10:22:50.184+0530    INFO    stateresolver/stateresolver.go:48       New State ID is pexNjfQP
2021-05-31T10:22:50.184+0530    INFO    stateresolver/stateresolver.go:49       Converging state requires execution of 1 step(s)
2021-05-31T10:22:50.332+0530    INFO    operation/operator.go:259       operation 'operation-install' skipped for fleet-server.7.13.0
2021-05-31T10:22:51.301+0530    INFO    log/reporter.go:40      2021-05-31T10:22:51+05:30 - message: Application: fleet-server--7.13.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-05-31T10:22:51.303+0530    INFO    stateresolver/stateresolver.go:66       Updating internal state
2021-05-31T10:22:51.795+0530    INFO    cmd/enroll_cmd.go:643   Fleet Server - Starting
2021-05-31T10:22:51.883+0530    ERROR   status/reporter.go:236  Elastic Agent status changed to: 'error'
2021-05-31T10:22:51.883+0530    ERROR   log/reporter.go:36      2021-05-31T10:22:51+05:30 - message: Application: fleet-server--7.13.0[]: State changed to CRASHED: exited with code: 1 - type: 'ERROR' - sub_type: 'FAILED'
2021-05-31T10:22:51.884+0530    INFO    status/reporter.go:236  Elastic Agent status changed to: 'online'
2021-05-31T10:22:51.884+0530    INFO    log/reporter.go:40      2021-05-31T10:22:51+05:30 - message: Application: fleet-server--7.13.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-05-31T10:22:51.884+0530    INFO    log/reporter.go:40      2021-05-31T10:22:51+05:30 - message: Application: fleet-server--7.13.0[]: State changed to RESTARTING: Restarting - type: 'STATE' - sub_type: 'STARTING'
2021-05-31T10:22:52.455+0530    INFO    log/reporter.go:40      2021-05-31T10:22:52+05:30 - message: Application: fleet-server--7.13.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-05-31T10:22:52.464+0530    ERROR   status/reporter.go:236  Elastic Agent status changed to: 'error'
2021-05-31T10:22:52.464+0530    ERROR   log/reporter.go:36      2021-05-31T10:22:52+05:30 - message: Application: fleet-server--7.13.0[]: State changed to CRASHED: exited with code: 1 - type: 'ERROR' - sub_type: 'FAILED'
2021-05-31T10:22:52.464+0530    INFO    status/reporter.go:236  Elastic Agent status changed to: 'online'
2021-05-31T10:22:52.465+0530    INFO    log/reporter.go:40      2021-05-31T10:22:52+05:30 - message: Application: fleet-server--7.13.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-05-31T10:22:52.465+0530    INFO    log/reporter.go:40      2021-05-31T10:22:52+05:30 - message: Application: fleet-server--7.13.0[]: State changed to RESTARTING: Restarting - type: 'STATE' - sub_type: 'STARTING'
2021-05-31T10:22:52.797+0530    INFO    cmd/enroll_cmd.go:643   Fleet Server - Restarting
2021-05-31T10:22:53.036+0530    INFO    log/reporter.go:40      2021-05-31T10:22:53+05:30 - message: Application: fleet-server--7.13.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-05-31T10:22:53.045+0530    ERROR   status/reporter.go:236  Elastic Agent status changed to: 'error'
2021-05-31T10:22:53.045+0530    ERROR   log/reporter.go:36      2021-05-31T10:22:53+05:30 - message: Application: fleet-server--7.13.0[]: State changed to CRASHED: exited with code: 1 - type: 'ERROR' - sub_type: 'FAILED'
2021-05-31T10:22:53.045+0530    INFO    status/reporter.go:236  Elastic Agent status changed to: 'online'
2021-05-31T10:22:53.046+0530    INFO    log/reporter.go:40      2021-05-31T10:22:53+05:30 - message: Application: fleet-server--7.13.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-05-31T10:22:53.046+0530    INFO    log/reporter.go:40      2021-05-31T10:22:53+05:30 - message: Application: fleet-server--7.13.0[]: State changed to RESTARTING: Restarting - type: 'STATE' - sub_type: 'STARTING'
2021-05-31T10:22:53.616+0530    INFO    log/reporter.go:40      2021-05-31T10:22:53+05:30 - message: Application: fleet-server--7.13.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-05-31T10:24:48.803+0530    INFO    log/reporter.go:40      2021-05-31T10:24:48+05:30 - message: Application: fleet-server--7.13.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-05-31T10:24:48.804+0530    INFO    log/reporter.go:40      2021-05-31T10:24:48+05:30 - message: Application: fleet-server--7.13.0[]: State changed to RESTARTING: Restarting - type: 'STATE' - sub_type: 'STARTING'
2021-05-31T10:24:49.044+0530    INFO    cmd/run.go:189  Shutting down Elastic Agent and sending last events...
2021-05-31T10:24:49.044+0530    INFO    operation/operator.go:191       waiting for installer of pipeline 'default' to finish
2021-05-31T10:24:49.044+0530    INFO    process/app.go:176      Signaling application to stop because of shutdown: fleet-server--7.13.0
2021-05-31T10:24:49.374+0530    INFO    log/reporter.go:40      2021-05-31T10:24:49+05:30 - message: Application: fleet-server--7.13.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-05-31T10:25:19.061+0530    INFO    cmd/run.go:197  Shutting down completed.
2021-05-31T10:25:19.061+0530    INFO    log/reporter.go:40      2021-05-31T10:25:19+05:30 - message: Application: fleet-server--7.13.0[]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'
2021-05-31T10:25:19.061+0530    INFO    [api]   api/server.go:66        Stats endpoint (/var/lib/elastic-agent/data/tmp/elastic-agent.sock) finished: accept unix /var/lib/elastic-agent/data/tmp/elastic-agent.sock: use of closed network connection
Error: fleet-server never started by elastic-agent daemon: context canceled

I had to reduce the repetitive stuff from the logs because it was not letting me post it here

Could you share the log files you find under data/elastic-agent-{hash}/logs? There should be a log file for fleet-server and elastic-agent.

Could you also share the command (redacted) which you use to start the elastic-agent?

sudo elastic-agent enroll -f --fleet-server-es=https://10.10.10.7:9200 --fleet-server-service-token=AAEAAWVsYXN0aWMvZmx2VydmVyL3Rva2VuLTE2MjIzNjQ1NjAxMzQ6cXJ5ak9tdEpRVTZBYV9sdGtwT1dfZw --fleet-server-es-ca /etc/elasticsearch/elastic-stack-ca.p12

{"log.level":"info","@timestamp":"2021-05-31T16:30:04.190+0530","log.origin":{"file.name":"warn/warn.go","file.line":18},"message":"The Elastic Agent is currently in BETA and should not be used in production","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:04.191+0530","log.origin":{"file.name":"application/application.go","file.line":68},"message":"Detecting execution mode","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:04.193+0530","log.origin":{"file.name":"application/application.go","file.line":89},"message":"Agent is in Fleet Server bootstrap mode","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:04.894+0530","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":62},"message":"Starting stats endpoint","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:04.894+0530","log.origin":{"file.name":"application/fleet_server_bootstrap.go","file.line":124},"message":"Agent is starting","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:04.894+0530","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":64},"message":"Metrics endpoint listening on: /var/lib/elastic-agent/data/tmp/elastic-agent.sock (configured: unix:///var/lib/elastic-agent/data/tmp/elastic-agent.sock)","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:04.899+0530","log.origin":{"file.name":"application/fleet_server_bootstrap.go","file.line":134},"message":"Agent is stopped","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:04.907+0530","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":48},"message":"New State ID is Z3MMr4gO","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:04.907+0530","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":49},"message":"Converging state requires execution of 1 step(s)","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:05.063+0530","log.origin":{"file.name":"operation/operator.go","file.line":259},"message":"operation 'operation-install' skipped for fleet-server.7.13.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:05.481+0530","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-05-31T16:30:05+05:30 - message: Application: fleet-server--7.13.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:05.484+0530","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":66},"message":"Updating internal state","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-05-31T16:30:06.062+0530","log.origin":{"file.name":"status/reporter.go","file.line":236},"message":"Elastic Agent status changed to: 'error'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-05-31T16:30:06.062+0530","log.origin":{"file.name":"log/reporter.go","file.line":36},"message":"2021-05-31T16:30:06+05:30 - message: Application: fleet-server--7.13.0[]: State changed to CRASHED: exited with code: 1 - type: 'ERROR' - sub_type: 'FAILED'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:06.063+0530","log.origin":{"file.name":"status/reporter.go","file.line":236},"message":"Elastic Agent status changed to: 'online'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:06.063+0530","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-05-31T16:30:06+05:30 - message: Application: fleet-server--7.13.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:06.063+0530","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-05-31T16:30:06+05:30 - message: Application: fleet-server--7.13.0[]: State changed to RESTARTING: Restarting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:06.636+0530","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-05-31T16:30:06+05:30 - message: Application: fleet-server--7.13.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-05-31T16:30:06.645+0530","log.origin":{"file.name":"status/reporter.go","file.line":236},"message":"Elastic Agent status changed to: 'error'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-05-31T16:30:06.645+0530","log.origin":{"file.name":"log/reporter.go","file.line":36},"message":"2021-05-31T16:30:06+05:30 - message: Application: fleet-server--7.13.0[]: State changed to CRASHED: exited with code: 1 - type: 'ERROR' - sub_type: 'FAILED'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:06.645+0530","log.origin":{"file.name":"status/reporter.go","file.line":236},"message":"Elastic Agent status changed to: 'online'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:06.646+0530","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-05-31T16:30:06+05:30 - message: Application: fleet-server--7.13.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:06.646+0530","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-05-31T16:30:06+05:30 - message: Application: fleet-server--7.13.0[]: State changed to RESTARTING: Restarting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:07.216+0530","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-05-31T16:30:07+05:30 - message: Application: fleet-server--7.13.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-05-31T16:30:07.225+0530","log.origin":{"file.name":"status/reporter.go","file.line":236},"message":"Elastic Agent status changed to: 'error'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-05-31T16:30:07.225+0530","log.origin":{"file.name":"log/reporter.go","file.line":36},"message":"2021-05-31T16:30:07+05:30 - message: Application: fleet-server--7.13.0[]: State changed to CRASHED: exited with code: 1 - type: 'ERROR' - sub_type: 'FAILED'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:07.226+0530","log.origin":{"file.name":"status/reporter.go","file.line":236},"message":"Elastic Agent status changed to: 'online'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:07.226+0530","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-05-31T16:30:07+05:30 - message: Application: fleet-server--7.13.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:07.226+0530","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-05-31T16:30:07+05:30 - message: Application: fleet-server--7.13.0[]: State changed to RESTARTING: Restarting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:07.798+0530","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-05-31T16:30:07+05:30 - message: Application: fleet-server--7.13.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-05-31T16:30:07.806+0530","log.origin":{"file.name":"status/reporter.go","file.line":236},"message":"Elastic Agent status changed to: 'error'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-05-31T16:30:07.806+0530","log.origin":{"file.name":"log/reporter.go","file.line":36},"message":"2021-05-31T16:30:07+05:30 - message: Application: fleet-server--7.13.0[]: State changed to CRASHED: exited with code: 1 - type: 'ERROR' - sub_type: 'FAILED'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:07.807+0530","log.origin":{"file.name":"status/reporter.go","file.line":236},"message":"Elastic Agent status changed to: 'online'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:07.807+0530","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-05-31T16:30:07+05:30 - message: Application: fleet-server--7.13.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:07.808+0530","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-05-31T16:30:07+05:30 - message: Application: fleet-server--7.13.0[]: State changed to RESTARTING: Restarting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:08.379+0530","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-05-31T16:30:08+05:30 - message: Application: fleet-server--7.13.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-05-31T16:30:08.385+0530","log.origin":{"file.name":"status/reporter.go","file.line":236},"message":"Elastic Agent status changed to: 'error'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-05-31T16:30:08.385+0530","log.origin":{"file.name":"log/reporter.go","file.line":36},"message":"2021-05-31T16:30:08+05:30 - message: Application: fleet-server--7.13.0[]: State changed to CRASHED: exited with code: 1 - type: 'ERROR' - sub_type: 'FAILED'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:08.386+0530","log.origin":{"file.name":"status/reporter.go","file.line":236},"message":"Elastic Agent status changed to: 'online'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:08.386+0530","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-05-31T16:30:08+05:30 - message: Application: fleet-server--7.13.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:08.386+0530","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-05-31T16:30:08+05:30 - message: Application: fleet-server--7.13.0[]: State changed to RESTARTING: Restarting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:08.949+0530","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-05-31T16:30:08+05:30 - message: Application: fleet-server--7.13.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-05-31T16:30:08.955+0530","log.origin":{"file.name":"status/reporter.go","file.line":236},"message":"Elastic Agent status changed to: 'error'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-05-31T16:30:08.956+0530","log.origin":{"file.name":"log/reporter.go","file.line":36},"message":"2021-05-31T16:30:08+05:30 - message: Application: fleet-server--7.13.0[]: State changed to CRASHED: exited with code: 1 - type: 'ERROR' - sub_type: 'FAILED'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:08.956+0530","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-05-31T16:30:08+05:30 - message: Application: fleet-server--7.13.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:32:04.941+0530","log.origin":{"file.name":"cmd/run.go","file.line":189},"message":"Shutting down Elastic Agent and sending last events...","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:32:04.941+0530","log.origin":{"file.name":"operation/operator.go","file.line":191},"message":"waiting for installer of pipeline 'default' to finish","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:32:04.941+0530","log.origin":{"file.name":"process/app.go","file.line":176},"message":"Signaling application to stop because of shutdown: fleet-server--7.13.0","ecs.version":"1.6.0"}

Thanks for the additional details. The log file you shared above looks like the elatsic-agent log. Could you check if you find the fleet-server log file too?

Nope. I don’t see any fleet server logs is my command for enrolling the agent correct?

I think I miss something obvious here. @michalp any ideas?

Being that the Fleet Server is crashing, I would check that port 8220 is free and that the host can communicate with elasticsearch from that location.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.