Hello. Last Thursday I upgraded an Elastic cluster to 7.17.10. Since then, Fleet has not been collecting any data from agents and the Fleet server itself doesn't appear to communicating with the cluster.
After restarting elastic-agent on the Fleet server, here are the logs:
2023-06-21T16:05:21.302-0400 INFO application/application.go:67 Detecting execution mode
2023-06-21T16:05:21.303-0400 INFO application/application.go:88 Agent is in Fleet Server bootstrap mode
2023-06-21T16:05:21.575-0400 INFO [api] api/server.go:62 Starting stats endpoint
2023-06-21T16:05:21.575-0400 INFO application/fleet_server_bootstrap.go:130 Agent is starting
2023-06-21T16:05:21.575-0400 INFO [api] api/server.go:64 Metrics endpoint listening on: /var/lib/elastic-agent/data/tmp/elastic-agent.sock (configured: unix:///var/lib/elastic-agent/data/tmp/elastic-agent.sock)
2023-06-21T16:05:21.575-0400 INFO application/fleet_server_bootstrap.go:140 Agent is stopped
2023-06-21T16:05:21.578-0400 INFO stateresolver/stateresolver.go:48 New State ID is rb9pAWf7
2023-06-21T16:05:21.578-0400 INFO stateresolver/stateresolver.go:49 Converging state requires execution of 1 step(s)
2023-06-21T16:05:21.612-0400 INFO operation/operator.go:284 operation 'operation-install' skipped for fleet-server.7.17.10
2023-06-21T16:05:21.694-0400 INFO log/reporter.go:40 2023-06-21T16:05:21-04:00 - message: Application: fleet-server--7.17.10[]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
2023-06-21T16:05:21.694-0400 INFO stateresolver/stateresolver.go:66 Updating internal state
2023-06-21T16:05:23.215-0400 WARN status/reporter.go:236 Elastic Agent status changed to: 'degraded'
2023-06-21T16:05:23.215-0400 INFO log/reporter.go:40 2023-06-21T16:05:23-04:00 - message: Application: fleet-server--7.17.10[]: State changed to DEGRADED: Running on policy with Fleet Server integration: 6b4fd400-1fb8-11ec-977d-05d1caf13f05; missing config fleet.agent.id (expected during bootstrap process) - type: 'STATE' - sub_type: 'RUNNING'
Agents do appear as "Healthy" in Fleet in Kibana and I am able to upgrade some of them to the newest version, though some get stuck in Updating phase.
The actual Fleet agent on the Default Fleet Server Policy has been offline since the upgrade.
elastic-agent is also in active (running) state on its server.
Wasn't sure the appropriate method for fixing this.