Unable to upgrade Fleet Server past 8.5.3

Elastic Stack Elastic Agent
1

I've been stuck trying to upgrade our Fleet Server past version 8.5.3. Every time I push the update manually (via curl) or through Kibana, I get some random errors (see below) and it reverts back to 8.5.3. For clarification, I've tried upgrading to 8.6.0, 8.6.1, 8.7.0, 8.8.0, 8.9.0, and 8.9.1. All had the same result.

Upgrade logs from elastic-agent:

10:42:56.610
elastic_agent
[elastic_agent][info] Upgrading agent
10:42:56.610
elastic_agent
[elastic_agent][info] 2023-10-12T10:42:56-05:00 - message: Application: [b80fc84c-d84f-4ec8-a8f7-ec201c6613cf]: State changed to UPDATING: Update to version '8.6.0' started - type: 'STATE' - sub_type: 'UPDATING'
10:43:03.357
elastic_agent
[elastic_agent][info] download from https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.6.0-linux-x86_64.tar.gz completed in 6 seconds @ 62.62MBps
10:43:03.483
elastic_agent
[elastic_agent][info] download from https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.6.0-linux-x86_64.tar.gz.sha512 completed in Less than a second @ +InfYBps
10:43:19.445
elastic_agent
[elastic_agent][info] Unpacked upgrade artifact
10:43:19.447
elastic_agent
[elastic_agent][info] Changing symlink
10:43:19.456
elastic_agent
[elastic_agent][info] Writing upgrade marker file
10:43:19.456
elastic_agent
[elastic_agent][info] Updating active commit
10:43:19.593
elastic_agent
[elastic_agent][info] Restarting after upgrade
10:43:19.593
elastic_agent
[elastic_agent][info] reexec Shutdown channel triggered
10:43:19.593
elastic_agent
[elastic_agent][info] Restarting Elastic Agent
10:43:19.593
elastic_agent
[elastic_agent][info] waiting for installer of pipeline 'default' to finish
10:43:19.593
elastic_agent
[elastic_agent][info] Signaling application to stop because of shutdown: filebeat--8.5.3--36643631373035623733363936343635
10:43:19.593
elastic_agent
[elastic_agent][info] Signaling application to stop because of shutdown: filebeat--8.5.3
10:43:19.593
elastic_agent
[elastic_agent][info] Signaling application to stop because of shutdown: fleet-server--8.5.3
10:43:19.966
elastic_agent
[elastic_agent][info] 2023-10-12T10:43:19-05:00 - message: Application: fleet-server--8.5.3[b80fc84c-d84f-4ec8-a8f7-ec201c6613cf]: State changed to STOPPING: Stopping - type: 'STATE' - sub_type: 'STOPPING'
10:43:19.966
elastic_agent
[elastic_agent][info] 2023-10-12T10:43:19-05:00 - message: Application: fleet-server--8.5.3[b80fc84c-d84f-4ec8-a8f7-ec201c6613cf]: State changed to STOPPING: Stopping - type: 'STATE' - sub_type: 'STOPPING'
10:43:20.096
elastic_agent
[elastic_agent][error] filebeat_monitoring stderr: "panic: close of closed channel\n\ngoroutine 86 [running]:\ngithub.com/elastic/beats/v7/filebeat/beater.(*Filebeat).Stop(0xc0007820c0)\n\tgithub.com/elastic/beats/v7/filebeat/beater/filebeat.go:435 +0x3f\ngithub.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch.func5()\n\tgithub.com/elastic/beats/v7/libbeat/cmd/instance/beat.go:462 +0x55\nsync.(*Once).doSlow(0x0?, 0x0?)\n\tsync/once.go:68 +0xc2\nsync.(*Once).Do(...)\n\tsync/once.go:59\ngithub.com/elastic/elastic-agent-libs/service.HandleSignals.func1()\n\tgithub.com/elastic/elastic-agent-libs@v0.2.11/service/service.go:60 +0x19c\ncreated by github.com/elastic/elastic-agent-libs/service.HandleSignals\n\tgithub.com/elastic/elastic-agent-libs@v0.2.11/service/service.go:49 +0x18f\n"
10:43:20.100
elastic_agent
[elastic_agent][error] filebeat stderr: "panic: close of closed channel\n\ngoroutine 105 [running]:\ngithub.com/elastic/beats/v7/filebeat/beater.(*Filebeat).Stop(0xc000568e70)\n\tgithub.com/elastic/beats/v7/filebeat/beater/filebeat.go:435 +0x3f\ngithub.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch.func5()\n\tgithub.com/elastic/beats/v7/libbeat/cmd/instance/beat.go:462 +0x55\nsync.(*Once).doSlow(0x0?, 0x0?)\n\tsync/once.go:68 +0xc2\nsync.(*Once).Do(...)\n\tsync/once.go:59\ngithub.com/elastic/elastic-agent-libs/service.HandleSignals.func1()\n\tgithub.com/elastic/elastic-agent-libs@v0.2.11/service/service.go:60 +0x19c\ncreated by github.com/elastic/elastic-agent-libs/service.HandleSignals\n\tgithub.com/elastic/elastic-agent-libs@v0.2.11/service/service.go:49 +0x18f\n"
10:43:20.105
elastic_agent
[elastic_agent][info] 2023-10-12T10:43:20-05:00 - message: Application: filebeat--8.5.3[b80fc84c-d84f-4ec8-a8f7-ec201c6613cf]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'
10:43:20.106
elastic_agent
[elastic_agent][info] 2023-10-12T10:43:20-05:00 - message: Application: filebeat--8.5.3--36643631373035623733363936343635[b80fc84c-d84f-4ec8-a8f7-ec201c6613cf]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'
10:43:20.595
elastic_agent
[elastic_agent][error] failed to stop fleet-server: os: process already finished
10:43:21.595
elastic_agent
[elastic_agent][info] 2023-10-12T10:43:21-05:00 - message: Application: fleet-server--8.5.3[b80fc84c-d84f-4ec8-a8f7-ec201c6613cf]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'
10:43:21.595
elastic_agent
[elastic_agent][info] Agent is stopped
10:43:21.943
elastic_agent
[elastic_agent][info] APM instrumentation disabled
10:43:21.945
elastic_agent
[elastic_agent][info] Gathered system information
10:43:21.965
elastic_agent
[elastic_agent][info] Detected available inputs and outputs
10:43:21.965
elastic_agent
[elastic_agent][info] Capabilities file not found in /opt/Elastic/Agent/capabilities.yml
10:43:21.965
elastic_agent
[elastic_agent][info] Determined allowed capabilities
10:43:22.116
elastic_agent
[elastic_agent][info] Parsed configuration and determined agent is managed by Fleet
10:43:22.116
elastic_agent
[elastic_agent][warn] SSL/TLS verifications disabled.
10:43:22.136
elastic_agent
[elastic_agent][info] Starting stats endpoint
10:43:22.137
elastic_agent
[elastic_agent][info] Metrics endpoint listening on: [::]:6791 (configured: http://:6791)
10:43:22.138
elastic_agent
[elastic_agent][info] Docker provider skipped, unable to connect: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
10:43:22.357
elastic_agent
[elastic_agent][warn] SSL/TLS verifications disabled.
10:43:22.358
elastic_agent
[elastic_agent][error] lazy acker: failed ack batch, enqueue for retry: [action_id: 5e4717c0-164f-4796-b78f-608993e41bab, type: UPGRADE]
10:43:22.358
elastic_agent
[elastic_agent][info] restoring current policy from disk
10:43:22.408
elastic_agent
[elastic_agent][info] Source URI changed from "https://artifacts.elastic.co/downloads/" to "https://artifacts.elastic.co/downloads/"
10:43:22.409
elastic_agent
[elastic_agent][info] Updating running component model
10:43:23.303
elastic_agent
[elastic_agent][error] Spawned new component fleet-server-default: output not supported
10:43:23.303
elastic_agent
[elastic_agent][error] Spawned new unit fleet-server-default-fleet-server-fleet_server-733ec415-9070-4d8a-8cf2-4b6b58dd697f: output not supported
10:43:23.303
elastic_agent
[elastic_agent][error] Spawned new unit fleet-server-default: output not supported
10:43:23.507
elastic_agent
[elastic_agent][info] Spawned new component log-default: Starting: spawned pid '22426'
10:43:23.507
elastic_agent
[elastic_agent][info] Spawned new unit log-default-logfile-system-94035698-6ee3-4987-a588-5c91182a72be: Starting: spawned pid '22426'
10:43:23.508
elastic_agent
[elastic_agent][info] Spawned new unit log-default: Starting: spawned pid '22426'
10:43:23.957
elastic_agent
[elastic_agent][info] Spawned new component filestream-monitoring: Starting: spawned pid '22431'
10:43:23.958
elastic_agent
[elastic_agent][info] Spawned new unit filestream-monitoring-filestream-monitoring-agent: Starting: spawned pid '22431'
10:43:23.958
elastic_agent
[elastic_agent][info] Spawned new unit filestream-monitoring: Starting: spawned pid '22431'
10:43:23.960
elastic_agent
[elastic_agent][error] lazy acker: failed ack batch, enqueue for retry: [action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]
10:43:23.960
elastic_agent
[elastic_agent][info] Updating running component model
10:43:24.187
elastic_agent
[elastic_agent][info] Component state changed log-default (STARTING->HEALTHY): Healthy: communicating with pid '22426'
10:43:24.202
elastic_agent
[elastic_agent][info] Component state changed filestream-monitoring (STARTING->HEALTHY): Healthy: communicating with pid '22431'
10:43:27.309
elastic_agent
[elastic_agent][info] Unit state changed filestream-monitoring-filestream-monitoring-agent (STARTING->HEALTHY): Healthy
10:43:27.309
elastic_agent
[elastic_agent][info] Unit state changed filestream-monitoring (STARTING->HEALTHY): Healthy
10:43:28.308
elastic_agent
[elastic_agent][info] Unit state changed log-default (STARTING->HEALTHY): Healthy
10:43:28.308
elastic_agent
[elastic_agent][info] Unit state changed log-default-logfile-system-94035698-6ee3-4987-a588-5c91182a72be (STARTING->HEALTHY): Healthy
10:43:59.992
elastic_agent
[elastic_agent][info] APM instrumentation disabled
10:43:59.998
elastic_agent
[elastic_agent][info] Detecting execution mode
10:44:00.019
elastic_agent
[elastic_agent][info] Agent is managed by Fleet
10:44:00.019
elastic_agent
[elastic_agent][info] capabilities file not found in /opt/Elastic/Agent/capabilities.yml
10:44:00.019
elastic_agent
[elastic_agent][warn] SSL/TLS verifications disabled.
10:44:00.438
elastic_agent
[elastic_agent][info] Docker provider skipped, unable to connect: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
10:44:00.473
elastic_agent
[elastic_agent][info] restoring current policy from disk
10:44:00.482
elastic_agent
[elastic_agent][info] Source URI changed from "https://artifacts.elastic.co/downloads/" to "https://artifacts.elastic.co/downloads/"
10:44:00.483
elastic_agent
[elastic_agent][info] New State ID is _W9Jxpjj
10:44:00.483
elastic_agent
[elastic_agent][info] Converging state requires execution of 3 step(s)
10:44:00.854
elastic_agent
[elastic_agent][info] download from https://artifacts.elastic.co/downloads/fleet-server/fleet-server-8.5.3-linux-x86_64.tar.gz completed in Less than a second @ +InfYBps
10:44:00.874
elastic_agent
[elastic_agent][info] download from https://artifacts.elastic.co/downloads/fleet-server/fleet-server-8.5.3-linux-x86_64.tar.gz.sha512 completed in Less than a second @ +InfYBps
10:44:00.874
elastic_agent
[elastic_agent][info] downloaded binary 'fleet-server.8.5.3' into '/opt/Elastic/Agent/data/elastic-agent-0e1a73/downloads/fleet-server-8.5.3-linux-x86_64.tar.gz' as part of operation 'operation-fetch'
10:44:01.264
elastic_agent
[elastic_agent][info] operation 'operation-install' skipped for fleet-server.8.5.3
10:44:01.551
elastic_agent
[elastic_agent][info] 2023-10-12T10:44:01-05:00 - message: Application: fleet-server--8.5.3[b80fc84c-d84f-4ec8-a8f7-ec201c6613cf]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
10:44:01.636
elastic_agent
[elastic_agent][error] fleet-server stderr: "{\"level\":\"info\",\"time\":\"2023-10-12T10:44:01-05:00\",\"message\":\"No applicable limit for 0 agents, using default.\"}\n"
10:44:01.639
elastic_agent
[elastic_agent][error] fleet-server stderr: "{\"level\":\"info\",\"time\":\"2023-10-12T10:44:01-05:00\",\"message\":\"No applicable limit for 0 agents, using default.\"}\n"
10:44:02.367
elastic_agent
[elastic_agent][info] download from https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.5.3-linux-x86_64.tar.gz completed in Less than a second @ +InfYBps
10:44:02.385
elastic_agent
[elastic_agent][info] download from https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.5.3-linux-x86_64.tar.gz.sha512 completed in Less than a second @ +InfYBps
10:44:02.385
elastic_agent
[elastic_agent][info] downloaded binary 'filebeat.8.5.3' into '/opt/Elastic/Agent/data/elastic-agent-0e1a73/downloads/filebeat-8.5.3-linux-x86_64.tar.gz' as part of operation 'operation-fetch'
10:44:03.160
elastic_agent
[elastic_agent][info] 2023-10-12T10:44:03-05:00 - message: Application: fleet-server--8.5.3[b80fc84c-d84f-4ec8-a8f7-ec201c6613cf]: State changed to RUNNING: Running on policy with Fleet Server integration: fleet-server-policy - type: 'STATE' - sub_type: 'RUNNING'
10:44:03.196
elastic_agent
[elastic_agent][info] operation 'operation-install' skipped for filebeat.8.5.3
10:44:03.391
elastic_agent
[elastic_agent][info] 2023-10-12T10:44:03-05:00 - message: Application: filebeat--8.5.3[b80fc84c-d84f-4ec8-a8f7-ec201c6613cf]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
10:44:03.873
elastic_agent
[elastic_agent][info] operation 'operation-install' skipped for filebeat.8.5.3
10:44:04.292
elastic_agent
[elastic_agent][info] 2023-10-12T10:44:04-05:00 - message: Application: filebeat--8.5.3--36643631373035623733363936343635[b80fc84c-d84f-4ec8-a8f7-ec201c6613cf]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
10:44:04.294
elastic_agent
[elastic_agent][info] Updating internal state
10:44:04.295
elastic_agent
[elastic_agent][warn] SSL/TLS verifications disabled.
10:44:04.298
elastic_agent
[elastic_agent][info] Source URI changed from "https://artifacts.elastic.co/downloads/" to "https://artifacts.elastic.co/downloads/"
10:44:04.298
elastic_agent
[elastic_agent][info] Source URI in operator changed to "https://artifacts.elastic.co/downloads/"
10:44:04.299
elastic_agent
[elastic_agent][info] New State ID is _W9Jxpjj
10:44:04.299
elastic_agent
[elastic_agent][info] Converging state requires execution of 2 step(s)
10:44:04.567
elastic_agent
[elastic_agent][info] Starting stats endpoint
10:44:04.567
elastic_agent
[elastic_agent][info] Agent is starting
10:44:04.568
elastic_agent
[elastic_agent][info] Metrics endpoint listening on: /opt/Elastic/Agent/data/tmp/elastic-agent.sock (configured: unix:///opt/Elastic/Agent/data/tmp/elastic-agent.sock)
10:44:04.853
elastic_agent
[elastic_agent][info] operation 'operation-install' skipped for filebeat.8.5.3
10:44:04.853
elastic_agent
[elastic_agent][info] operation 'operation-start' skipped for filebeat.8.5.3
10:44:05.304
elastic_agent
[elastic_agent][info] operation 'operation-install' skipped for filebeat.8.5.3
10:44:05.304
elastic_agent
[elastic_agent][info] operation 'operation-start' skipped for filebeat.8.5.3
10:44:05.305
elastic_agent
[elastic_agent][info] Updating internal state
10:44:05.488
elastic_agent
[elastic_agent][info] 2023-10-12T10:44:05-05:00 - message: Application: filebeat--8.5.3--36643631373035623733363936343635[b80fc84c-d84f-4ec8-a8f7-ec201c6613cf]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'
10:44:07.794
elastic_agent
[elastic_agent][info] 2023-10-12T10:44:07-05:00 - message: Application: filebeat--8.5.3[b80fc84c-d84f-4ec8-a8f7-ec201c6613cf]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'10:42:56.610
elastic_agent
[elastic_agent][info] Upgrading agent
10:42:56.610
elastic_agent
[elastic_agent][info] 2023-10-12T10:42:56-05:00 - message: Application: [b80fc84c-d84f-4ec8-a8f7-ec201c6613cf]: State changed to UPDATING: Update to version '8.6.0' started - type: 'STATE' - sub_type: 'UPDATING'
10:43:03.357
elastic_agent
[elastic_agent][info] download from https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.6.0-linux-x86_64.tar.gz completed in 6 seconds @ 62.62MBps
10:43:03.483
elastic_agent
[elastic_agent][info] download from https://artifacts.elastic.co/downloads/beats/elastic-agent/elastic-agent-8.6.0-linux-x86_64.tar.gz.sha512 completed in Less than a second @ +InfYBps
10:43:19.445
elastic_agent
[elastic_agent][info] Unpacked upgrade artifact
10:43:19.447
elastic_agent
[elastic_agent][info] Changing symlink
10:43:19.456
elastic_agent
[elastic_agent][info] Writing upgrade marker file
10:43:19.456
elastic_agent
[elastic_agent][info] Updating active commit
10:43:19.593
elastic_agent
[elastic_agent][info] Restarting after upgrade
10:43:19.593
elastic_agent
[elastic_agent][info] reexec Shutdown channel triggered
10:43:19.593
elastic_agent
[elastic_agent][info] Restarting Elastic Agent
10:43:19.593
elastic_agent
[elastic_agent][info] waiting for installer of pipeline 'default' to finish
10:43:19.593
elastic_agent
[elastic_agent][info] Signaling application to stop because of shutdown: filebeat--8.5.3--36643631373035623733363936343635
10:43:19.593
elastic_agent
[elastic_agent][info] Signaling application to stop because of shutdown: filebeat--8.5.3
10:43:19.593
elastic_agent
[elastic_agent][info] Signaling application to stop because of shutdown: fleet-server--8.5.3
10:43:19.966
elastic_agent
[elastic_agent][info] 2023-10-12T10:43:19-05:00 - message: Application: fleet-server--8.5.3[b80fc84c-d84f-4ec8-a8f7-ec201c6613cf]: State changed to STOPPING: Stopping - type: 'STATE' - sub_type: 'STOPPING'
10:43:19.966
elastic_agent
[elastic_agent][info] 2023-10-12T10:43:19-05:00 - message: Application: fleet-server--8.5.3[b80fc84c-d84f-4ec8-a8f7-ec201c6613cf]: State changed to STOPPING: Stopping - type: 'STATE' - sub_type: 'STOPPING'
10:43:20.096
elastic_agent
[elastic_agent][error] filebeat_monitoring stderr: "panic: close of closed channel\n\ngoroutine 86 [running]:\ngithub.com/elastic/beats/v7/filebeat/beater.(*Filebeat).Stop(0xc0007820c0)\n\tgithub.com/elastic/beats/v7/filebeat/beater/filebeat.go:435 +0x3f\ngithub.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch.func5()\n\tgithub.com/elastic/beats/v7/libbeat/cmd/instance/beat.go:462 +0x55\nsync.(*Once).doSlow(0x0?, 0x0?)\n\tsync/once.go:68 +0xc2\nsync.(*Once).Do(...)\n\tsync/once.go:59\ngithub.com/elastic/elastic-agent-libs/service.HandleSignals.func1()\n\tgithub.com/elastic/elastic-agent-libs@v0.2.11/service/service.go:60 +0x19c\ncreated by github.com/elastic/elastic-agent-libs/service.HandleSignals\n\tgithub.com/elastic/elastic-agent-libs@v0.2.11/service/service.go:49 +0x18f\n"
10:43:20.100
elastic_agent
[elastic_agent][error] filebeat stderr: "panic: close of closed channel\n\ngoroutine 105 [running]:\ngithub.com/elastic/beats/v7/filebeat/beater.(*Filebeat).Stop(0xc000568e70)\n\tgithub.com/elastic/beats/v7/filebeat/beater/filebeat.go:435 +0x3f\ngithub.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch.func5()\n\tgithub.com/elastic/beats/v7/libbeat/cmd/instance/beat.go:462 +0x55\nsync.(*Once).doSlow(0x0?, 0x0?)\n\tsync/once.go:68 +0xc2\nsync.(*Once).Do(...)\n\tsync/once.go:59\ngithub.com/elastic/elastic-agent-libs/service.HandleSignals.func1()\n\tgithub.com/elastic/elastic-agent-libs@v0.2.11/service/service.go:60 +0x19c\ncreated by github.com/elastic/elastic-agent-libs/service.HandleSignals\n\tgithub.com/elastic/elastic-agent-libs@v0.2.11/service/service.go:49 +0x18f\n"
10:43:20.105
elastic_agent
[elastic_agent][info] 2023-10-12T10:43:20-05:00 - message: Application: filebeat--8.5.3[b80fc84c-d84f-4ec8-a8f7-ec201c6613cf]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'
10:43:20.106
elastic_agent
[elastic_agent][info] 2023-10-12T10:43:20-05:00 - message: Application: filebeat--8.5.3--36643631373035623733363936343635[b80fc84c-d84f-4ec8-a8f7-ec201c6613cf]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'
10:43:20.595
elastic_agent
[elastic_agent][error] failed to stop fleet-server: os: process already finished
10:43:21.595
elastic_agent
[elastic_agent][info] 2023-10-12T10:43:21-05:00 - message: Application: fleet-server--8.5.3[b80fc84c-d84f-4ec8-a8f7-ec201c6613cf]: State changed to STOPPED: Stopped - type: 'STATE' - sub_type: 'STOPPED'
10:43:21.595
elastic_agent
[elastic_agent][info] Agent is stopped
10:43:21.943
elastic_agent
[elastic_agent][info] APM instrumentation disabled
10:43:21.945
elastic_agent
[elastic_agent][info] Gathered system information
10:43:21.965
elastic_agent
[elastic_agent][info] Detected available inputs and outputs
10:43:21.965
elastic_agent
[elastic_agent][info] Capabilities file not found in /opt/Elastic/Agent/capabilities.yml
10:43:21.965
elastic_agent
[elastic_agent][info] Determined allowed capabilities
10:43:22.116
elastic_agent
[elastic_agent][info] Parsed configuration and determined agent is managed by Fleet
10:43:22.116
elastic_agent
[elastic_agent][warn] SSL/TLS verifications disabled.
10:43:22.136
elastic_agent
[elastic_agent][info] Starting stats endpoint
10:43:22.137
elastic_agent
[elastic_agent][info] Metrics endpoint listening on: [::]:6791 (configured: http://:6791)
10:43:22.138
elastic_agent
[elastic_agent][info] Docker provider skipped, unable to connect: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
10:43:22.357
elastic_agent
[elastic_agent][warn] SSL/TLS verifications disabled.
10:43:22.358
elastic_agent
[elastic_agent][error] lazy acker: failed ack batch, enqueue for retry: [action_id: 5e4717c0-164f-4796-b78f-608993e41bab, type: UPGRADE]
10:43:22.358
elastic_agent
[elastic_agent][info] restoring current policy from disk
10:43:22.408
elastic_agent
[elastic_agent][info] Source URI changed from "https://artifacts.elastic.co/downloads/" to "https://artifacts.elastic.co/downloads/"
10:43:22.409
elastic_agent
[elastic_agent][info] Updating running component model
10:43:23.303
elastic_agent
[elastic_agent][error] Spawned new component fleet-server-default: output not supported
10:43:23.303
elastic_agent
[elastic_agent][error] Spawned new unit fleet-server-default-fleet-server-fleet_server-733ec415-9070-4d8a-8cf2-4b6b58dd697f: output not supported
10:43:23.303
elastic_agent
[elastic_agent][error] Spawned new unit fleet-server-default: output not supported
10:43:23.507
elastic_agent
[elastic_agent][info] Spawned new component log-default: Starting: spawned pid '22426'
10:43:23.507
elastic_agent
[elastic_agent][info] Spawned new unit log-default-logfile-system-94035698-6ee3-4987-a588-5c91182a72be: Starting: spawned pid '22426'
10:43:23.508
elastic_agent
[elastic_agent][info] Spawned new unit log-default: Starting: spawned pid '22426'
10:43:23.957
elastic_agent
[elastic_agent][info] Spawned new component filestream-monitoring: Starting: spawned pid '22431'
10:43:23.958
elastic_agent
[elastic_agent][info] Spawned new unit filestream-monitoring-filestream-monitoring-agent: Starting: spawned pid '22431'
10:43:23.958
elastic_agent
[elastic_agent][info] Spawned new unit filestream-monitoring: Starting: spawned pid '22431'
10:43:23.960
elastic_agent
[elastic_agent][error] lazy acker: failed ack batch, enqueue for retry: [action_id: policy:fleet-server-policy:17:1, type: POLICY_CHANGE]
10:43:23.960
elastic_agent
[elastic_agent][info] Updating running component model
10:43:24.187
elastic_agent
[elastic_agent][info] Component state changed log-default (STARTING->HEALTHY): Healthy: communicating with pid '22426'
10:43:24.202
elastic_agent
[elastic_agent][info] Component state changed filestream-monitoring (STARTING->HEALTHY): Healthy: communicating with pid '22431'
10:43:27.309
elastic_agent
[elastic_agent][info] Unit state changed filestream-monitoring-filestream-monitoring-agent (STARTING->HEALTHY): Healthy
10:43:27.309
elastic_agent
[elastic_agent][info] Unit state changed filestream-monitoring (STARTING->HEALTHY): Healthy
10:43:28.308
elastic_agent
[elastic_agent][info] Unit state changed log-default (STARTING->HEALTHY): Healthy
10:43:28.308
elastic_agent
[elastic_agent][info] Unit state changed log-default-logfile-system-94035698-6ee3-4987-a588-5c91182a72be (STARTING->HEALTHY): Healthy
10:43:59.992
elastic_agent
[elastic_agent][info] APM instrumentation disabled
10:43:59.998
elastic_agent
[elastic_agent][info] Detecting execution mode
10:44:00.019
elastic_agent
[elastic_agent][info] Agent is managed by Fleet
10:44:00.019
elastic_agent
[elastic_agent][info] capabilities file not found in /opt/Elastic/Agent/capabilities.yml
10:44:00.019
elastic_agent
[elastic_agent][warn] SSL/TLS verifications disabled.
10:44:00.438
elastic_agent
[elastic_agent][info] Docker provider skipped, unable to connect: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
10:44:00.473
elastic_agent
[elastic_agent][info] restoring current policy from disk
10:44:00.482
elastic_agent
[elastic_agent][info] Source URI changed from "https://artifacts.elastic.co/downloads/" to "https://artifacts.elastic.co/downloads/"
10:44:00.483
elastic_agent
[elastic_agent][info] New State ID is _W9Jxpjj
10:44:00.483
elastic_agent
[elastic_agent][info] Converging state requires execution of 3 step(s)
10:44:00.854
elastic_agent
[elastic_agent][info] download from https://artifacts.elastic.co/downloads/fleet-server/fleet-server-8.5.3-linux-x86_64.tar.gz completed in Less than a second @ +InfYBps
10:44:00.874
elastic_agent
[elastic_agent][info] download from https://artifacts.elastic.co/downloads/fleet-server/fleet-server-8.5.3-linux-x86_64.tar.gz.sha512 completed in Less than a second @ +InfYBps
10:44:00.874
elastic_agent
[elastic_agent][info] downloaded binary 'fleet-server.8.5.3' into '/opt/Elastic/Agent/data/elastic-agent-0e1a73/downloads/fleet-server-8.5.3-linux-x86_64.tar.gz' as part of operation 'operation-fetch'
10:44:01.264
elastic_agent
[elastic_agent][info] operation 'operation-install' skipped for fleet-server.8.5.3
10:44:01.551
elastic_agent
[elastic_agent][info] 2023-10-12T10:44:01-05:00 - message: Application: fleet-server--8.5.3[b80fc84c-d84f-4ec8-a8f7-ec201c6613cf]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
10:44:01.636
elastic_agent
[elastic_agent][error] fleet-server stderr: "{\"level\":\"info\",\"time\":\"2023-10-12T10:44:01-05:00\",\"message\":\"No applicable limit for 0 agents, using default.\"}\n"
10:44:01.639
elastic_agent
[elastic_agent][error] fleet-server stderr: "{\"level\":\"info\",\"time\":\"2023-10-12T10:44:01-05:00\",\"message\":\"No applicable limit for 0 agents, using default.\"}\n"
10:44:02.367
elastic_agent
[elastic_agent][info] download from https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.5.3-linux-x86_64.tar.gz completed in Less than a second @ +InfYBps
10:44:02.385
elastic_agent
[elastic_agent][info] download from https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-8.5.3-linux-x86_64.tar.gz.sha512 completed in Less than a second @ +InfYBps
10:44:02.385
elastic_agent
[elastic_agent][info] downloaded binary 'filebeat.8.5.3' into '/opt/Elastic/Agent/data/elastic-agent-0e1a73/downloads/filebeat-8.5.3-linux-x86_64.tar.gz' as part of operation 'operation-fetch'
10:44:03.160
elastic_agent
[elastic_agent][info] 2023-10-12T10:44:03-05:00 - message: Application: fleet-server--8.5.3[b80fc84c-d84f-4ec8-a8f7-ec201c6613cf]: State changed to RUNNING: Running on policy with Fleet Server integration: fleet-server-policy - type: 'STATE' - sub_type: 'RUNNING'
10:44:03.196
elastic_agent
[elastic_agent][info] operation 'operation-install' skipped for filebeat.8.5.3
10:44:03.391
elastic_agent
[elastic_agent][info] 2023-10-12T10:44:03-05:00 - message: Application: filebeat--8.5.3[b80fc84c-d84f-4ec8-a8f7-ec201c6613cf]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
10:44:03.873
elastic_agent
[elastic_agent][info] operation 'operation-install' skipped for filebeat.8.5.3
10:44:04.292
elastic_agent
[elastic_agent][info] 2023-10-12T10:44:04-05:00 - message: Application: filebeat--8.5.3--36643631373035623733363936343635[b80fc84c-d84f-4ec8-a8f7-ec201c6613cf]: State changed to STARTING: Starting - type: 'STATE' - sub_type: 'STARTING'
10:44:04.294
elastic_agent
[elastic_agent][info] Updating internal state
10:44:04.295
elastic_agent
[elastic_agent][warn] SSL/TLS verifications disabled.
10:44:04.298
elastic_agent
[elastic_agent][info] Source URI changed from "https://artifacts.elastic.co/downloads/" to "https://artifacts.elastic.co/downloads/"
10:44:04.298
elastic_agent
[elastic_agent][info] Source URI in operator changed to "https://artifacts.elastic.co/downloads/"
10:44:04.299
elastic_agent
[elastic_agent][info] New State ID is _W9Jxpjj
10:44:04.299
elastic_agent
[elastic_agent][info] Converging state requires execution of 2 step(s)
10:44:04.567
elastic_agent
[elastic_agent][info] Starting stats endpoint
10:44:04.567
elastic_agent
[elastic_agent][info] Agent is starting
10:44:04.568
elastic_agent
[elastic_agent][info] Metrics endpoint listening on: /opt/Elastic/Agent/data/tmp/elastic-agent.sock (configured: unix:///opt/Elastic/Agent/data/tmp/elastic-agent.sock)
10:44:04.853
elastic_agent
[elastic_agent][info] operation 'operation-install' skipped for filebeat.8.5.3
10:44:04.853
elastic_agent
[elastic_agent][info] operation 'operation-start' skipped for filebeat.8.5.3
10:44:05.304
elastic_agent
[elastic_agent][info] operation 'operation-install' skipped for filebeat.8.5.3
10:44:05.304
elastic_agent
[elastic_agent][info] operation 'operation-start' skipped for filebeat.8.5.3
10:44:05.305
elastic_agent
[elastic_agent][info] Updating internal state
10:44:05.488
elastic_agent
[elastic_agent][info] 2023-10-12T10:44:05-05:00 - message: Application: filebeat--8.5.3--36643631373035623733363936343635[b80fc84c-d84f-4ec8-a8f7-ec201c6613cf]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'
10:44:07.794
elastic_agent
[elastic_agent][info] 2023-10-12T10:44:07-05:00 - message: Application: filebeat--8.5.3[b80fc84c-d84f-4ec8-a8f7-ec201c6613cf]: State changed to RUNNING: Running - type: 'STATE' - sub_type: 'RUNNING'

./elastic-agent status --output=yaml:

./elastic-agent status --output=yaml
status: 2
message: ""
applications:
- id: fleet-server--8.5.3-31effa43
  name: fleet-server
  status: 2
  message: 'Running on policy with Fleet Server integration: fleet-server-policy'
  payload: {}
- id: filebeat--8.5.3-d8e5b528
  name: filebeat
  status: 2
  message: Running
  payload: {}
- id: filebeat--8.5.3--36643631373035623733363936343635-c268cdd1
  name: filebeat_monitoring
  status: 2
  message: Running
  payload: {}

GET .fleet-agents/_doc/b80fc84c-d84f-4ec8-a8f7-ec201c6613cf:

#! this request accesses system indices: [.fleet-agents-7], but in a future major version, direct access to system indices will be prevented by default
{
  "_index": ".fleet-agents-7",
  "_id": "b80fc84c-d84f-4ec8-a8f7-ec201c6613cf",
  "_version": 1048105,
  "_seq_no": 88552120,
  "_primary_term": 15,
  "found": true,
  "_source": {
    "policy_coordinator_idx": 2,
    "outputs": {
      "default": {
        "api_key": "",
        "permissions_hash": "b499bb1fc918edd3374a11ca3914b23118844bf427e33b2d4cb8d2901c8e9891",
        "type": "elasticsearch",
        "to_retire_api_key_ids": [
          {
            "retired_at": "2023-10-04T15:10:49Z",
            "id": "uSCmFIQBZANMB30rZhh8"
          }
        ],
        "api_key_id": ""
      }
    },
    "default_api_key": null,
    "agent": {
      "id": "b80fc84c-d84f-4ec8-a8f7-ec201c6613cf",
      "version": "8.5.3"
    },
    "enrolled_at": "2022-10-24T14:53:04Z",
    "policy_id": "fleet-server-policy",
    "last_checkin_status": "online",
    "active": true,
    "local_metadata": {
      "os": {
        "kernel": "5.14.21-150500.55.19-default",
        "name": "openSUSE Leap",
        "family": "",
        "version": "15.5",
        "platform": "opensuse-leap",
        "full": "openSUSE Leap(15.5)"
      },
      "elastic": {
        "agent": {
          "build.original": "8.5.3 (build: 0e1a7396f356f8af5ad28c3c8e941256f50c86a2 at 2022-12-06 00:04:15 +0000 UTC)",
          "upgradeable": true,
          "log_level": "info",
          "id": "b80fc84c-d84f-4ec8-a8f7-ec201c6613cf",
          "version": "8.5.3",
          "snapshot": false
        }
      },
      "host": {
        "hostname": "Fett",
        "ip": [
          "127.0.0.1/8",
          "::1/128",
          "172.19.4.228/24",
          "fe80::250:56ff:fe91:254b/64"
        ],
        "name": "Fett",
        "id": "f1f4346b13cbaa9959aade336331c439",
        "mac": [
          "00:50:56:91:25:4b"
        ],
        "architecture": "x86_64"
      }
    },
    "type": "PERMANENT",
    "last_checkin": "2023-10-12T15:31:48Z",
    "upgrade_started_at": "2023-10-12T15:18:07.968Z",
    "access_api_key_id": "QB97CoQBZANMB30rnqqD",
    "default_api_key_id": null,
    "updated_at": "2023-10-12T15:31:57Z",
    "policy_revision_idx": 17,
    "policy_output_permissions_hash": null,
    "action_seq_no": [
      82
    ],
    "upgraded_at": null
  }
}

Not sure what else to try. Send help.

2

Still having this issue. Any thoughts?

3

In which version is the rest of the stack?

You mentioned all those version, but what is the version of Elasticsearch and Kibana that you are running?

4

The rest of the stack (Kibana, ES, and Logstash) is on 8.9.1.

5

How did you installed the fleet server? Using the tar.gz package or the rpm/deb packages?

You are trying to upgrade through the Fleet UI on Kibana, right?

6

How did you installed the fleet server? Using the tar.gz package or the rpm/deb packages?

It's been a while since I've installed the fleet server, but I believe I used the 'tar.gz' package via the script that was given from the Fleet UI in Kibana during Fleet setup.

You are trying to upgrade through the Fleet UI on Kibana, right?

I've tried upgrading through the Fleet UI in Kibana as well as sending an API request via curl to force the upgrade.

7

You can confirm this checking if you have the package installed.

sudo rpm -qa | grep agent for RPM based and sudo apt list --installed | grep agent for DEB based.

Not sure what is the issue, the log is not very helpful, but I would try to upgrade through the command line.

sudo /opt/Elastic/Agent/elastic-agent upgrade 8.9.1

8

You can confirm this checking if you have the package installed.

sudo rpm -qa | grep agent for RPM based and sudo apt list --installed | grep agent for DEB based.

Confirmed that the package was not installed via rpm.

Not sure what is the issue, the log is not very helpful, but I would try to upgrade through the command line.

sudo /opt/Elastic/Agent/elastic-agent upgrade 8.9.1

Upgrading via the command line was unsuccessful. Here's the output:
Error: Failed trigger upgrade of daemon: cannot be upgraded; perform upgrading using Fleet

I checked the agent status and all is listed as "healthy". Let me know if there's anything else I can check to help solve this issue.

9

I have no idea what could be the cause, I recommend that you open an issue on Github or wait to see if someone from Elastic can provide further help.

10

Thanks for trying. Hopefully someone from Elastic can help.

11

Still having issues. Any ideas?

12

This looks strange, can you provide the policy?

13

Here's the fleet server policy.

id: fleet-server-policy
revision: 18
outputs:
  default:
    type: logstash
    hosts:
      - '172.19.4.47:5044'
      - '172.19.4.174:5044'
      - '172.19.4.180:5044'
    ssl:
      certificate: |-
        -----BEGIN CERTIFICATE-----
        MIIDPjCCAiagAwIBAgIUGSAi9ikw3ERshFCYvrgdCja1T08wDQYJKoZIhvcNAQEL
        BQAwNDEyMDAGA1UEAxMpRWxhc3RpYyBDZXJ0aWZpY2F0ZSBUb29sIEF1dG9nZW5l
        cmF0ZWQgQ0EwHhcNMjIxMDI2MjEyNzIzWhcNMjUxMDI1MjEyNzIzWjARMQ8wDQYD
        VQQDEwZjbGllbnQwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDXetX9
        0IRPtNXxeyORLWJEbY3D0QFnzx+Xgbm4qrupO0VUO18fG6DupjkdaWRsVqZJErCC
        dlKBNG5ymEUrZ8wWDtsm6gPkeQuZxG5GzaR8io9XJeiujB2sloDL5clrIqy+s/ut
        FFY3QpsL5g+sgLTqae1oO1yEgaI918qUIlKwAgesnvUTmdsAPMRefge3dnz/kson
        DkoJZvO+jFNIE6ycttSS/UHjsNy/w3ole5GKPLCPHs4dIXhg/iQstTitQ4elwyq5
        N0xAnJK+kBlvJYwLhSiUuA3LDDDW8hjHb9+4armBB9IKHBUmFOkSCB7ZYy6gqaSj
        WjcQjqSMP9WDNFN3AgMBAAGjazBpMB0GA1UdDgQWBBRORycMg3APlUjJkb64mz6Z
        2+40tzAfBgNVHSMEGDAWgBSKe6wmDeoLGEvmN2gB69fLHiihSTAcBgNVHREEFTAT
        ghFmZXR0Lm1zdW5ldDJrLmVkdTAJBgNVHRMEAjAAMA0GCSqGSIb3DQEBCwUAA4IB
        AQCakhf1o3vIxHtXQyIf8mYt8kMYdgx649GaxKtGkYxaaWo6U/8LbO4h6XkWW/7Y
        m2MSB3BLjYhBWznHwaXpEmUOCl77/zpM4wKpUrgu4MLkxRqUUOARzymLj+0HK00o
        DHctDvmtXnR4AV55KBLB40rD41/kI1Ak+GvGv9lMxbINKYFKmFBiU/+hinJiARJp
        2I9wMmuDJQhwHk1E6IlLyuCnUXL9HI/5fVEq0kMi+ZPnXnaa9W9yziYApHklLhEA
        3/TdX8WwZCiXvDxk1XyJWy0AaJcRVE+HdKZ7I1ILkbwWbnawxChzUoig+QasS9Ge
        39sytQVqt7MbazpXYzmeHWet
        -----END CERTIFICATE-----
      key: |
        -----BEGIN RSA PRIVATE KEY-----
        MIIEpQIBAAKCAQEA13rV/dCET7TV8XsjkS1iRG2Nw9EBZ88fl4G5uKq7qTtFVDtf
        Hxug7qY5HWlkbFamSRKwgnZSgTRucphFK2fMFg7bJuoD5HkLmcRuRs2kfIqPVyXo
        rowdrJaAy+XJayKsvrP7rRRWN0KbC+YPrIC06mntaDtchIGiPdfKlCJSsAIHrJ71
        E5nbADzEXn4Ht3Z8/5LKJw5KCWbzvoxTSBOsnLbUkv1B47Dcv8N6JXuRijywjx7O
        HSF4YP4kLLU4rUOHpcMquTdMQJySvpAZbyWMC4UolLgNywww1vIYx2/fuGq5gQfS
        ChwVJhTpEgge2WMuoKmko1o3EI6kjD/VgzRTdwIDAQABAoIBAARzG5V7h3/t3WJ4
        hyoppGyCpqQd3h2QAA2s4XJCyaVGUrmD5a71bxlT2BIXaboi/+66XWgURsKMRqDv
        LAkoF5AiYRF9+56YMU53k/S6V7nf/UhwwUv5cL4Y+DKTCC8XhDcGmHKL3PaUgDHQ
        CmV8Mure98a96GcuwMmYzyzH7f/ca89bmaLfce825gXpLDgNV1al5MbH+q/jDOf/
        ZxCouOeT4F6NMO+YREgb5RBLPz5uB3OlO2dnvZ2D/hoylJC9cohMcEwDmZwJzdt8
        J5ZtVjgayoJtDKAG0jHfHHtRsSTyMeREBDHfb7Q4EqO3XyrFlDjIkQn3H8IgJa2i
        mqQkPgECgYEA/QY3ullQWWnb3jaQx/Cv/3b9N6pW3rJVqXjx+Hph+YIIija/dVmY
        0HkqQRDJ/LvVws7S/55yC9OprGw0CzHoqo5G1xRChvAFcFMx1dHxRNVZ5rTm4e4G
        3BP4VlrfU4CoRMR1WjMq9RXRQPnryDe/JAHj9lVZLnAQvpHVeg6D+bcCgYEA2gOV
        MnDwkr1QheiFL2ckbiZ9uHG6a54dwDy3iXOxZ2HpESb7DvnZxhak2vfh15Qmww/e
        BRr4HyLXsVj3cvhTb0ylU053HKZ0eXFLBNQDOdLbUIZdFDQj7vy9CNZWMq+wAwad
        eoJET4uj95UU+64cSXrO655uK3LbwpU5H2nNdEECgYEAjiZ3Go9g78rkRnRbZKjZ
        B+n+ms5tHZINRDf5oUKIVpfyWkwlxjT/AGiCD6nyALY6gsaVjhyjL/fLRF5J4tw4
        PdHPBLiSI6MnOxLSVbEsHiLK7i1F5TpHyk7B3yoQe6Y1YuUIlRNYF/cY56tBBuwi
        y4TwYI1UnB1DmvztdTRAewMCgYEAl4W380fVnFchmQAjWO/ZvQjZhhDdHSDW0p4g
        fdK/8Pd20ioCgcUbG3bvexSfjYJ1cksb5vZ4jsNbn/ISgYCgFJHzsbOuS2ZRO3qH
        RkLIPy8SqKPKG1mlw9s60uKNJEtBDMlTOCtDqPmHq3KdE9NN67AEEnSZY28/17x4
        6CFT30ECgYEA8Ug9ucWe0sIGfZBBGiqj3BNCQI9irkpeu5VrNvjvGpvkZtDZlpvQ
        2zG/vXYytJQrz32cHqbC/JO6KZySR0UvTT1s20vnrNYs2zJfM7LhzPvcYxSu8i9o
        FpelivDN2dfU5Q2qLgX6pGUO5Aso78ko/LkJQ86YFoilE4NgrcMpEf4=
        -----END RSA PRIVATE KEY-----
      certificate_authorities:
        - |-
          -----BEGIN CERTIFICATE-----
          MIIDSTCCAjGgAwIBAgIUXsIjmqXHNm1kAWAMA696e7V+Y+swDQYJKoZIhvcNAQEL
          BQAwNDEyMDAGA1UEAxMpRWxhc3RpYyBDZXJ0aWZpY2F0ZSBUb29sIEF1dG9nZW5l
          cmF0ZWQgQ0EwHhcNMjIxMDI2MjEyMjM5WhcNMjUxMDI1MjEyMjM5WjA0MTIwMAYD
          VQQDEylFbGFzdGljIENlcnRpZmljYXRlIFRvb2wgQXV0b2dlbmVyYXRlZCBDQTCC
          ASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKa7g+zMHd2NhAURZHCsLkti
          YoynEHdgR8vD5yRZ7BE9LJ/W3RwPArHGNTayDAiX9U7qu5DALHZ2Kj1DWnhEtGd0
          nyYPvGdB60r0z8EVK/pwlmx9GlvVAe0Owpe0L42wqxSRDwT/smXhzyyvVAZdypTX
          L+y64jVmMMezoTf7vV2AHu3Kiz91z9xX1+WFVWpNPaL4eNxXd+gTPblfhagEy8Iz
          XpVqEXIGDhU5M/DZiN2bTfHq99T/wBUgq9ihrP5Bv/STW5bczPB0OzthTi178fa9
          OO3XhHHG4KwIz50tENYm3Uyl4XRPWntVg+tL80Cj5KxAAt3amLrb5rfl0CLEwisC
          AwEAAaNTMFEwHQYDVR0OBBYEFIp7rCYN6gsYS+Y3aAHr18seKKFJMB8GA1UdIwQY
          MBaAFIp7rCYN6gsYS+Y3aAHr18seKKFJMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZI
          hvcNAQELBQADggEBAKJSpOd9F5KlPLvMMcya6nlrrzTVVs7fO1FUaDusqX1d542I
          InuS3pOyfj7sfVSuKsftdseBRN3w1IHdB3TkTZgIdVQZY3kxfKfOnzE4OQoIgqcR
          4ySfMUX5OBIkFHSRo/qv0HowqAM5YU2rpW86yQ+kKcVfhBr21Qnb4QAanSJUqVvZ
          IpRrYSoQQC9VIEt44yX0MNdfsvffIYooD7y/qSQugE++r7TxXFwW2iFM73EKY09f
          9xjprDrwb35FVykmEcm81USxHzJ7haGYrwvN3EfUMKkLw/0ejHGhbkITp4rK+O6P
          koYRx5gE+dCMzuj2jZdXsBLD6lL2QJQoxZQzbn4=
          -----END CERTIFICATE-----
fleet:
  hosts:
    - 'https://172.19.4.228:8220'
output_permissions: {}
agent:
  download:
    sourceURI: 'https://artifacts.elastic.co/downloads/'
  monitoring:
    enabled: true
    use_output: default
    namespace: default
    logs: true
    metrics: false
  features: {}
  protection:
    enabled: false
    uninstall_token_hash: ''
    signing_key: >-
      MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAE31STDca28Gk2NqB/O+o6SmkfanFRQr+roBZ8gBEqeGr++d1cFV/CtqlU2vBGyB2JusyLNAYlUX/eu+mV3GCE5w==
inputs:
  - id: fleet-server-fleet_server-733ec415-9070-4d8a-8cf2-4b6b58dd697f
    name: fleet_server-1
    revision: 2
    type: fleet-server
    use_output: default
    meta:
      package:
        name: fleet_server
        version: 1.3.1
    data_stream:
      namespace: default
    package_policy_id: 733ec415-9070-4d8a-8cf2-4b6b58dd697f
    unused_key: not_used
  - id: logfile-system-94035698-6ee3-4987-a588-5c91182a72be
    name: system-1
    revision: 6
    type: logfile
    use_output: default
    meta:
      package:
        name: system
        version: 1.41.0
    data_stream:
      namespace: default
    package_policy_id: 94035698-6ee3-4987-a588-5c91182a72be
    streams:
      - id: logfile-system.auth-94035698-6ee3-4987-a588-5c91182a72be
        data_stream:
          dataset: system.auth
          type: logs
        exclude_files:
          - .gz$
        ignore_older: 72h
        paths:
          - /var/log/auth.log*
          - /var/log/secure*
        multiline:
          pattern: ^\s
          match: after
        processors:
          - add_locale: null
        tags:
          - hashicorp-vault-log
      - id: logfile-system.syslog-94035698-6ee3-4987-a588-5c91182a72be
        data_stream:
          dataset: system.syslog
          type: logs
        exclude_files:
          - .gz$
        ignore_older: 72h
        paths:
          - /var/log/messages*
          - /var/log/syslog*
        multiline:
          pattern: ^\s
          match: after
        processors:
          - add_locale: null
secret_references: []
signed:
  data: >-
    eyJpZCI6ImZsZWV0LXNlcnZlci1wb2xpY3kiLCJhZ2VudCI6eyJwcm90ZWN0aW9uIjp7ImVuYWJsZWQiOmZhbHNlLCJ1bmluc3RhbGxfdG9rZW5faGFzaCI6IiIsInNpZ25pbmdfa2V5IjoiTUZrd0V3WUhLb1pJemowQ0FRWUlLb1pJemowREFRY0RRZ0FFMzFTVERjYTI4R2syTnFCL08rbzZTbWtmYW5GUlFyK3JvQlo4Z0JFcWVHcisrZDFjRlYvQ3RxbFUydkJHeUIySnVzeUxOQVlsVVgvZXUrbVYzR0NFNXc9PSJ9fX0=
  signature: >-
    MEUCIGrAC9yrvNekkNkbHbzHYWNAimmcwpzLKwiv62W1U+VpAiEA0thZbxpOjWt/Af6xbDzFXHyIHvi1w67LHMvH7NEm4LA=
14

Oh, i think that this is the issue, the Fleet Server Integration does not support output to Logstash, only Elasticsearch.

If I'm not wrong this wasn't validate correctly before 8.6, but now it is, so this will prevent you from upgradring Fleet Server past 8.5.3, you will need to change the output of the policy that has the Fleet Server Integration to use Elasticsearch as an output.

Check this github issue about this.

15

That fixed it. Are there other integrations that won't accept logstash as an output? Would you recommend using Elasticsearch as the default for all agent policies?

Thank you so much for your help!

16

No, only the Fleet Server Integration does not support Logstash, the other integrations I don't think they had any issue.

It depends on your needs, I have a policy for the fleet server that uses the Elasticsearch output and all my other policies uses the Logstash output.

17

Gotcha. Thank you again!

18

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.