I have installed ECK and elasticsearch, kibana and fleet (agents and server) via the recipe yaml files in my kubernetes cluster running in Azure (version 1.21) and the fleet server keeps crashing without staying up. The manifests that I have applied are these ones: https://raw.githubusercontent.com/elastic/cloud-on-k8s/2.3/config/recipes/elastic-agent/fleet-kubernetes-integration.yaml
And the error while looking at the logs of the fleet server is this one:
{"log.level":"info","@timestamp":"2022-08-11T14:39:30.127Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-08-11T14:39:30Z - message: Application: fleet-server--8.3.3[]: State changed to STARTING: Waiting on default policy with Fleet Server integration - type: 'STATE' - sub_type: 'STARTING'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-11T14:39:38.021Z","log.origin":{"file.name":"cmd/enroll_cmd.go","file.line":792},"message":"Fleet Server - Waiting on default policy with Fleet Server integration","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-11T14:41:22.920Z","log.origin":{"file.name":"cmd/run.go","file.line":253},"message":"Shutting down Elastic Agent and sending last events...","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-11T14:41:22.920Z","log.origin":{"file.name":"operation/operator.go","file.line":222},"message":"waiting for installer of pipeline 'default' to finish","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-11T14:41:22.920Z","log.origin":{"file.name":"process/app.go","file.line":182},"message":"Signaling application to stop because of shutdown: fleet-server--8.3.3","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2022-08-11T14:41:23.420Z","log.origin":{"file.name":"process/app.go","file.line":290},"message":"failed to stop fleet-server: os: process already finished","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-11T14:41:24.421Z","log.origin":{"file.name":"log/reporter.go","file.line":40},"message":"2022-08-11T14:41:24Z - message: Application: fleet-server--8.3.3[]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-11T14:41:24.421Z","log.origin":{"file.name":"cmd/run.go","file.line":261},"message":"Shutting down completed.","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2022-08-11T14:41:24.421Z","log.logger":"api","log.origin":{"file.name":"api/server.go","file.line":72},"message":"Stats endpoint (/usr/share/elastic-agent/state/data/tmp/elastic-agent.sock) finished: accept unix /usr/share/elastic-agent/state/data/tmp/elastic-agent.sock: use of closed network connection","ecs.version":"1.6.0"}
Error: fleet-server failed: context canceled
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.3/fleet-troubleshooting.html
Error: enrollment failed: exit status 1
For help, please see our troubleshooting guide at https://www.elastic.co/guide/en/fleet/8.3/fleet-troubleshooting.html
The error shows that State changed to STARTING: Waiting on default policy with Fleet Server integration - type: 'STATE.
The kibana and elasticsearch are in green, all agents pods are running with no errors in their logs.
Besides this, the integrations that I have defined in xpack.fleet.packages: are not being shown either in the Kibana UI, check on the screenshot below, and there is nothing in the logs that can help me debugging this.
Is this a known bug/error, if so what's the action to solve it?
Thanks