Fleet Server keeps crashing

Ameer_Mukadam · May 31, 2021, 3:11am

So I wanted to explore the new fleet changes which where introduced lately I dod a complete new setup to get hands on with it. But I am not able to get the fleet server running it keeps crashing with exit code 1 and I am not getting any more information than this in the logs too.
Mine is a on prem setup so has anyone got the fleet server running in an on prem setup yet?

2021-05-31T10:22:46.776+0530    INFO    cmd/enroll_cmd.go:300   Generating self-signed certificate for Fleet Server
2021-05-31T10:22:48.786+0530    INFO    cmd/enroll_cmd.go:468   Spawning Elastic Agent daemon as a subprocess to complete bootstrap process.
2021-05-31T10:22:49.044+0530    INFO    warn/warn.go:18 The Elastic Agent is currently in BETA and should not be used in production
2021-05-31T10:22:49.045+0530    INFO    application/application.go:68   Detecting execution mode
2021-05-31T10:22:49.047+0530    INFO    application/application.go:89   Agent is in Fleet Server bootstrap mode
2021-05-31T10:22:49.791+0530    INFO    cmd/enroll_cmd.go:610   Waiting for Elastic Agent to start Fleet Server
2021-05-31T10:22:50.175+0530    INFO    [api]   api/server.go:62        Starting stats endpoint
2021-05-31T10:22:50.175+0530    INFO    application/fleet_server_bootstrap.go:124       Agent is starting
2021-05-31T10:22:50.176+0530    INFO    [api]   api/server.go:64        Metrics endpoint listening on: /var/lib/elastic-agent/data/tmp/elastic-agent.sock (configured: unix:///var/lib/elastic-agent/data/tmp/elastic-agent.sock)
2021-05-31T10:22:50.178+0530    INFO    application/fleet_server_bootstrap.go:134       Agent is stopped
2021-05-31T10:22:50.184+0530    INFO    stateresolver/stateresolver.go:48       New State ID is pexNjfQP
2021-05-31T10:22:50.184+0530    INFO    stateresolver/stateresolver.go:49       Converging state requires execution of 1 step(s)
2021-05-31T10:22:50.332+0530    INFO    operation/operator.go:259       operation 'operation-install' skipped for fleet-server.7.13.0
2021-05-31T10:22:51.301+0530    INFO    log/reporter.go:40      2021-05-31T10:22:51+05:30 - message: Application: fleet-server--7.13.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-05-31T10:22:51.303+0530    INFO    stateresolver/stateresolver.go:66       Updating internal state
2021-05-31T10:22:51.795+0530    INFO    cmd/enroll_cmd.go:643   Fleet Server - Starting
2021-05-31T10:22:51.883+0530    ERROR   status/reporter.go:236  Elastic Agent status changed to: 'error'
2021-05-31T10:22:51.883+0530    ERROR   log/reporter.go:36      2021-05-31T10:22:51+05:30 - message: Application: fleet-server--7.13.0[]: State changed to CRASHED: exited with code: 1 - type: 'ERROR' - sub_type: 'FAILED'
2021-05-31T10:22:51.884+0530    INFO    status/reporter.go:236  Elastic Agent status changed to: 'online'
2021-05-31T10:22:51.884+0530    INFO    log/reporter.go:40      2021-05-31T10:22:51+05:30 - message: Application: fleet-server--7.13.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-05-31T10:22:51.884+0530    INFO    log/reporter.go:40      2021-05-31T10:22:51+05:30 - message: Application: fleet-server--7.13.0[]: State changed to RESTARTING: Restarting - type: 'STATE' - sub_type: 'STARTING'
2021-05-31T10:22:52.455+0530    INFO    log/reporter.go:40      2021-05-31T10:22:52+05:30 - message: Application: fleet-server--7.13.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-05-31T10:22:52.464+0530    ERROR   status/reporter.go:236  Elastic Agent status changed to: 'error'
2021-05-31T10:22:52.464+0530    ERROR   log/reporter.go:36      2021-05-31T10:22:52+05:30 - message: Application: fleet-server--7.13.0[]: State changed to CRASHED: exited with code: 1 - type: 'ERROR' - sub_type: 'FAILED'
2021-05-31T10:22:52.464+0530    INFO    status/reporter.go:236  Elastic Agent status changed to: 'online'
2021-05-31T10:22:52.465+0530    INFO    log/reporter.go:40      2021-05-31T10:22:52+05:30 - message: Application: fleet-server--7.13.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-05-31T10:22:52.465+0530    INFO    log/reporter.go:40      2021-05-31T10:22:52+05:30 - message: Application: fleet-server--7.13.0[]: State changed to RESTARTING: Restarting - type: 'STATE' - sub_type: 'STARTING'
2021-05-31T10:22:52.797+0530    INFO    cmd/enroll_cmd.go:643   Fleet Server - Restarting
2021-05-31T10:22:53.036+0530    INFO    log/reporter.go:40      2021-05-31T10:22:53+05:30 - message: Application: fleet-server--7.13.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-05-31T10:22:53.045+0530    ERROR   status/reporter.go:236  Elastic Agent status changed to: 'error'
2021-05-31T10:22:53.045+0530    ERROR   log/reporter.go:36      2021-05-31T10:22:53+05:30 - message: Application: fleet-server--7.13.0[]: State changed to CRASHED: exited with code: 1 - type: 'ERROR' - sub_type: 'FAILED'
2021-05-31T10:22:53.045+0530    INFO    status/reporter.go:236  Elastic Agent status changed to: 'online'
2021-05-31T10:22:53.046+0530    INFO    log/reporter.go:40      2021-05-31T10:22:53+05:30 - message: Application: fleet-server--7.13.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-05-31T10:22:53.046+0530    INFO    log/reporter.go:40      2021-05-31T10:22:53+05:30 - message: Application: fleet-server--7.13.0[]: State changed to RESTARTING: Restarting - type: 'STATE' - sub_type: 'STARTING'
2021-05-31T10:22:53.616+0530    INFO    log/reporter.go:40      2021-05-31T10:22:53+05:30 - message: Application: fleet-server--7.13.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-05-31T10:24:48.803+0530    INFO    log/reporter.go:40      2021-05-31T10:24:48+05:30 - message: Application: fleet-server--7.13.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-05-31T10:24:48.804+0530    INFO    log/reporter.go:40      2021-05-31T10:24:48+05:30 - message: Application: fleet-server--7.13.0[]: State changed to RESTARTING: Restarting - type: 'STATE' - sub_type: 'STARTING'
2021-05-31T10:24:49.044+0530    INFO    cmd/run.go:189  Shutting down Elastic Agent and sending last events...
2021-05-31T10:24:49.044+0530    INFO    operation/operator.go:191       waiting for installer of pipeline 'default' to finish
2021-05-31T10:24:49.044+0530    INFO    process/app.go:176      Signaling application to stop because of shutdown: fleet-server--7.13.0
2021-05-31T10:24:49.374+0530    INFO    log/reporter.go:40      2021-05-31T10:24:49+05:30 - message: Application: fleet-server--7.13.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2021-05-31T10:25:19.061+0530    INFO    cmd/run.go:197  Shutting down completed.
2021-05-31T10:25:19.061+0530    INFO    log/reporter.go:40      2021-05-31T10:25:19+05:30 - message: Application: fleet-server--7.13.0[]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'
2021-05-31T10:25:19.061+0530    INFO    [api]   api/server.go:66        Stats endpoint (/var/lib/elastic-agent/data/tmp/elastic-agent.sock) finished: accept unix /var/lib/elastic-agent/data/tmp/elastic-agent.sock: use of closed network connection
Error: fleet-server never started by elastic-agent daemon: context canceled

I had to reduce the repetitive stuff from the logs because it was not letting me post it here

ruflin · May 31, 2021, 9:35am

Could you share the log files you find under data/elastic-agent-{hash}/logs? There should be a log file for fleet-server and elastic-agent.

Could you also share the command (redacted) which you use to start the elastic-agent?

Ameer_Mukadam · May 31, 2021, 11:06am

sudo elastic-agent enroll -f --fleet-server-es=https://10.10.10.7:9200 --fleet-server-service-token=AAEAAWVsYXN0aWMvZmx2VydmVyL3Rva2VuLTE2MjIzNjQ1NjAxMzQ6cXJ5ak9tdEpRVTZBYV9sdGtwT1dfZw --fleet-server-es-ca /etc/elasticsearch/elastic-stack-ca.p12

{"log.level":"info","@timestamp":"2021-05-31T16:30:04.190+0530","log.origin":{"file.name":"warn/warn.go","file.line":18},"message":"The Elastic Agent is currently in BETA and should not be used in production","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:04.191+0530","log.origin":{"file.name":"application/application.go","file.line":68},"message":"Detecting execution mode","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:04.193+0530","log.origin":{"file.name":"application/application.go","file.line":89},"message":"Agent is in Fleet Server bootstrap mode","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:04.894+0530","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":62},"message":"Starting stats endpoint","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:04.894+0530","log.origin":{"file.name":"application/fleet_server_bootstrap.go","file.line":124},"message":"Agent is starting","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:04.894+0530","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":64},"message":"Metrics endpoint listening on: /var/lib/elastic-agent/data/tmp/elastic-agent.sock (configured: unix:///var/lib/elastic-agent/data/tmp/elastic-agent.sock)","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:04.899+0530","log.origin":{"file.name":"application/fleet_server_bootstrap.go","file.line":134},"message":"Agent is stopped","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:04.907+0530","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":48},"message":"New State ID is Z3MMr4gO","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:04.907+0530","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":49},"message":"Converging state requires execution of 1 step(s)","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:05.063+0530","log.origin":{"file.name":"operation/operator.go","file.line":259},"message":"operation 'operation-install' skipped for fleet-server.7.13.0","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:05.481+0530","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-05-31T16:30:05+05:30 - message: Application: fleet-server--7.13.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:05.484+0530","log.origin":{"file.name":"stateresolver/stateresolver.go","file.line":66},"message":"Updating internal state","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-05-31T16:30:06.062+0530","log.origin":{"file.name":"status/reporter.go","file.line":236},"message":"Elastic Agent status changed to: 'error'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-05-31T16:30:06.062+0530","log.origin":{"file.name":"log/reporter.go","file.line":36},"message":"2021-05-31T16:30:06+05:30 - message: Application: fleet-server--7.13.0[]: State changed to CRASHED: exited with code: 1 - type: 'ERROR' - sub_type: 'FAILED'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:06.063+0530","log.origin":{"file.name":"status/reporter.go","file.line":236},"message":"Elastic Agent status changed to: 'online'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:06.063+0530","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-05-31T16:30:06+05:30 - message: Application: fleet-server--7.13.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:06.063+0530","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-05-31T16:30:06+05:30 - message: Application: fleet-server--7.13.0[]: State changed to RESTARTING: Restarting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:06.636+0530","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-05-31T16:30:06+05:30 - message: Application: fleet-server--7.13.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-05-31T16:30:06.645+0530","log.origin":{"file.name":"status/reporter.go","file.line":236},"message":"Elastic Agent status changed to: 'error'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-05-31T16:30:06.645+0530","log.origin":{"file.name":"log/reporter.go","file.line":36},"message":"2021-05-31T16:30:06+05:30 - message: Application: fleet-server--7.13.0[]: State changed to CRASHED: exited with code: 1 - type: 'ERROR' - sub_type: 'FAILED'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:06.645+0530","log.origin":{"file.name":"status/reporter.go","file.line":236},"message":"Elastic Agent status changed to: 'online'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:06.646+0530","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-05-31T16:30:06+05:30 - message: Application: fleet-server--7.13.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:06.646+0530","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-05-31T16:30:06+05:30 - message: Application: fleet-server--7.13.0[]: State changed to RESTARTING: Restarting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:07.216+0530","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-05-31T16:30:07+05:30 - message: Application: fleet-server--7.13.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-05-31T16:30:07.225+0530","log.origin":{"file.name":"status/reporter.go","file.line":236},"message":"Elastic Agent status changed to: 'error'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-05-31T16:30:07.225+0530","log.origin":{"file.name":"log/reporter.go","file.line":36},"message":"2021-05-31T16:30:07+05:30 - message: Application: fleet-server--7.13.0[]: State changed to CRASHED: exited with code: 1 - type: 'ERROR' - sub_type: 'FAILED'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:07.226+0530","log.origin":{"file.name":"status/reporter.go","file.line":236},"message":"Elastic Agent status changed to: 'online'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:07.226+0530","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-05-31T16:30:07+05:30 - message: Application: fleet-server--7.13.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:07.226+0530","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-05-31T16:30:07+05:30 - message: Application: fleet-server--7.13.0[]: State changed to RESTARTING: Restarting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:07.798+0530","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-05-31T16:30:07+05:30 - message: Application: fleet-server--7.13.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-05-31T16:30:07.806+0530","log.origin":{"file.name":"status/reporter.go","file.line":236},"message":"Elastic Agent status changed to: 'error'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-05-31T16:30:07.806+0530","log.origin":{"file.name":"log/reporter.go","file.line":36},"message":"2021-05-31T16:30:07+05:30 - message: Application: fleet-server--7.13.0[]: State changed to CRASHED: exited with code: 1 - type: 'ERROR' - sub_type: 'FAILED'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:07.807+0530","log.origin":{"file.name":"status/reporter.go","file.line":236},"message":"Elastic Agent status changed to: 'online'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:07.807+0530","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-05-31T16:30:07+05:30 - message: Application: fleet-server--7.13.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:07.808+0530","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-05-31T16:30:07+05:30 - message: Application: fleet-server--7.13.0[]: State changed to RESTARTING: Restarting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:08.379+0530","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-05-31T16:30:08+05:30 - message: Application: fleet-server--7.13.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-05-31T16:30:08.385+0530","log.origin":{"file.name":"status/reporter.go","file.line":236},"message":"Elastic Agent status changed to: 'error'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-05-31T16:30:08.385+0530","log.origin":{"file.name":"log/reporter.go","file.line":36},"message":"2021-05-31T16:30:08+05:30 - message: Application: fleet-server--7.13.0[]: State changed to CRASHED: exited with code: 1 - type: 'ERROR' - sub_type: 'FAILED'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:08.386+0530","log.origin":{"file.name":"status/reporter.go","file.line":236},"message":"Elastic Agent status changed to: 'online'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:08.386+0530","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-05-31T16:30:08+05:30 - message: Application: fleet-server--7.13.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:08.386+0530","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-05-31T16:30:08+05:30 - message: Application: fleet-server--7.13.0[]: State changed to RESTARTING: Restarting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:08.949+0530","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-05-31T16:30:08+05:30 - message: Application: fleet-server--7.13.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-05-31T16:30:08.955+0530","log.origin":{"file.name":"status/reporter.go","file.line":236},"message":"Elastic Agent status changed to: 'error'","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2021-05-31T16:30:08.956+0530","log.origin":{"file.name":"log/reporter.go","file.line":36},"message":"2021-05-31T16:30:08+05:30 - message: Application: fleet-server--7.13.0[]: State changed to CRASHED: exited with code: 1 - type: 'ERROR' - sub_type: 'FAILED'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:30:08.956+0530","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2021-05-31T16:30:08+05:30 - message: Application: fleet-server--7.13.0[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:32:04.941+0530","log.origin":{"file.name":"cmd/run.go","file.line":189},"message":"Shutting down Elastic Agent and sending last events...","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:32:04.941+0530","log.origin":{"file.name":"operation/operator.go","file.line":191},"message":"waiting for installer of pipeline 'default' to finish","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2021-05-31T16:32:04.941+0530","log.origin":{"file.name":"process/app.go","file.line":176},"message":"Signaling application to stop because of shutdown: fleet-server--7.13.0","ecs.version":"1.6.0"}

ruflin · June 1, 2021, 6:45am

Thanks for the additional details. The log file you shared above looks like the elatsic-agent log. Could you check if you find the fleet-server log file too?

Ameer_Mukadam · June 1, 2021, 7:55am

Nope. I don’t see any fleet server logs is my command for enrolling the agent correct?

ruflin · June 8, 2021, 2:10pm

I think I miss something obvious here. @michalp any ideas?

blaker · June 8, 2021, 2:54pm

Being that the Fleet Server is crashing, I would check that port 8220 is free and that the host can communicate with elasticsearch from that location.

system · July 6, 2021, 2:55pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.