# Elastic Agents fail to upgrade from Fleet, shows "Updating" status for more than a week
**URL:** https://discuss.elastic.co/t/elastic-agents-fail-to-upgrade-from-fleet-shows-updating-status-for-more-than-a-week/325426
**Category:** Elastic Agent
**Tags:** fleet
**Created:** [February 13, 2023, 9:48pm UTC](https://discuss.elastic.co/t/elastic-agents-fail-to-upgrade-from-fleet-shows-updating-status-for-more-than-a-week/325426 "2023-02-13T21:48:55Z")
**Posts on this page:** 15
**Page:** 1
### Author:  [@DougR](https://discuss.elastic.co/u/DougR)
#### Post date: [February 13, 2023, 9:48pm UTC](https://discuss.elastic.co/t/elastic-agents-fail-to-upgrade-from-fleet-shows-updating-status-for-more-than-a-week/325426/1 "2023-02-13T21:48:55Z")
I recently upgraded my Elastic Cloud instance to 8.6.1. After upgrading, I triggered an agent upgrade in Fleet to v8.6.1. The agents DID NOT upgrade and were stuck in `Updating` status for almost a week. If I go into the server and manually trigger an upgrade, the upgrade occur on the server, fleet will show the new version, but the Agent Status in Fleet will still show as `updating`. The command I use to do the upgrade in my Linux environment is:
```auto
sudo /opt/Elastic/Agent/elastic-agent upgrade 8.6.1
```
I found [another post](https://discuss.elastic.co/t/fleet-server-status-stuck-at-updating-but-seems-to-be-working-normally/318193) from December which reported a similar behavior, but I was unclear from the answers there whether following those instructions would remove the `updating` status from my agents, update Fleet instead (which is updated), or unenroll my agents.
The instructions in [that post](https://discuss.elastic.co/t/fleet-server-status-stuck-at-updating-but-seems-to-be-working-normally/318193) were:
> [@Fleet Server status stuck at updating, but seems to be working normally?!](https://discuss.elastic.co/t/fleet-server-status-stuck-at-updating-but-seems-to-be-working-normally/318193/2):
>
> Unfortunately, after upgrading, some agents can get stuck in "updating" state in Kibana, even if the actual underlying state is "healthy".
>
> To un-mark any Elastic Agent as upgrading, you can follow the below commands as superuser.
>
> Create a new service token:
>
> ```auto
>
> curl -X POST --user elastic:${SUPERUSER_PASS} -H 'x-elastic-product-origin:fleet' -H' content-type:application/json' "https://${ELASTICSEARCH_HOST}/_security/service/elastic/fleet-server/credential/token/fix-agents"
>
> ```
>
> Next, do an `update_by_query` . The query finds all Elastic Agents which are marked `updating` and marks the update as complete. With the created token replace the `${TOKEN}` below with that created token so the query can complete successfully:
>
> ```auto
>
> curl -XPOST -H' Authorization: Bearer ${TOKEN}' -H' x-elastic-product-origin:fleet' -H'content-type:application/json' "https://${ELASTICSEARCH_HOST}/.fleet-agents/_update_by_query" -d '{"query": {"bool": {"must": [{ "exists": { "field": "upgrade_started_at" } }],"must_not": [{ "exists": { "field": "upgraded_at" } }]}},"script": {"source": "ctx._source.upgraded_at = ctx._source.upgrade_started_at; ctx._source.upgrade_started_at = null;","lang": "painless"}}'
>
> ```
>
> Finally, clean up the created service token:
>
> ```auto
>
> curl -XDELETE --user elastic:${SUPERUSER_PASS} -H 'x-elastic-product-origin:fleet' -H'content-type:application/json' "https://${ELASTICSEARCH_HOST}/_security/service/elastic/fleet-server/credential/token/fix-agents"
>
> ```
>
> Note that this procedure will apply with a fleet servers as well, as they are a special case of agent.
In a later post in that discussion, the individual giving the answer wrote:
> [@Fleet Server status stuck at updating, but seems to be working normally?!](https://discuss.elastic.co/t/fleet-server-status-stuck-at-updating-but-seems-to-be-working-normally/318193/5):
>
> I re-read your question and I might have misunderstood your problem, sorry about that.
>
> I thought that you had agents in `updating` state, but I'm now realising that instead it's the Fleet Server state that it's stuck in "updating" state in the UI. Is it correct?
>
> Of course in this case disregards my previous response, since it would unenroll your existing agents.
>
> Would you mind to share some screenshots of what you see?
---
### Author:  [@Julia\_Bardi](https://discuss.elastic.co/u/Julia_Bardi)
#### Post date: [February 14, 2023, 9:38am UTC](https://discuss.elastic.co/t/elastic-agents-fail-to-upgrade-from-fleet-shows-updating-status-for-more-than-a-week/325426/2 "2023-02-14T09:38:44Z")
Hello,
So are agents actually upgraded just stuck in Updating state? It would help to check the Agent logs to see why it hasn't gone back to Healthy.
I think the linked commands should work to move back agents to Healthy manually, I don't see why agents would be unenrolled.
---
### Author:  [@DougR](https://discuss.elastic.co/u/DougR)
#### Post date: [February 14, 2023, 1:37pm UTC](https://discuss.elastic.co/t/elastic-agents-fail-to-upgrade-from-fleet-shows-updating-status-for-more-than-a-week/325426/3 "2023-02-14T13:37:52Z")
> [@Julia\_Bardi](#):
>
> So are agents actually upgraded just stuck in Updating state? It would help to check the Agent logs to see why it hasn't gone back to Healthy.
> I think the linked commands should work to move back agents to Healthy manually, I don't see why agents would be unenrolled.
No. The agents never updated. I can get them to update by going in and manually updating from the host (and the upgraded version shows in Fleet), but it still shows as updating in Fleet.
---
### Author:  [@Julia\_Bardi](https://discuss.elastic.co/u/Julia_Bardi)
#### Post date: [February 15, 2023, 10:22am UTC](https://discuss.elastic.co/t/elastic-agents-fail-to-upgrade-from-fleet-shows-updating-status-for-more-than-a-week/325426/4 "2023-02-15T10:22:50Z")
It would be best to check why the Upgrade action from Kibana doesn't work, you can gather the agent logs with `elastic-agent diagnostics collect` command.
---
### Author:  [@DougR](https://discuss.elastic.co/u/DougR)
#### Post date: [February 15, 2023, 5:37pm UTC](https://discuss.elastic.co/t/elastic-agents-fail-to-upgrade-from-fleet-shows-updating-status-for-more-than-a-week/325426/5 "2023-02-15T17:37:17Z")
I've pulled this. Where can I upload it to you, since it's rather large, and I'd rather not post it publicly, since I may have missed some redactions.
---
### Author:  [@Julia\_Bardi](https://discuss.elastic.co/u/Julia_Bardi)
#### Post date: [February 16, 2023, 7:53am UTC](https://discuss.elastic.co/t/elastic-agents-fail-to-upgrade-from-fleet-shows-updating-status-for-more-than-a-week/325426/6 "2023-02-16T07:53:47Z")
You can upload somewhere and share it with me: [julia.bardi@elastic.co](mailto:julia.bardi@elastic.co)
---
### Author:  [@DougR](https://discuss.elastic.co/u/DougR)
#### Post date: [February 16, 2023, 5:39pm UTC](https://discuss.elastic.co/t/elastic-agents-fail-to-upgrade-from-fleet-shows-updating-status-for-more-than-a-week/325426/7 "2023-02-16T17:39:42Z")
My problem is that I'm not able to upload it to anywhere like dropbox, box, google drive, etc.
---
### Author:  [@matled](https://discuss.elastic.co/u/matled)
#### Post date: [February 17, 2023, 1:36pm UTC](https://discuss.elastic.co/t/elastic-agents-fail-to-upgrade-from-fleet-shows-updating-status-for-more-than-a-week/325426/8 "2023-02-17T13:36:25Z")
Same issue here. Elastic-Stack 8.6.2. Upgraded from 8.6.1 to 8.6.2
We have two Fleet-Server
One went fine. The other one is stuck on Updating with several other Agents.
Log of Elastic-Fleet Server not beeing able to Upgrade and stuck in the upgrading state. Tried to restart the elastic agent service, still stuck
```auto
14:11:13.647[elastic_agent][info] signal "terminated" received
14:11:13.647[elastic_agent][info] Shutting down Elastic Agent and sending last events...
14:11:13.647[elastic_agent][info] signal "terminated" received
14:11:13.647[elastic_agent][info] Shutting down Elastic Agent and sending last events...
14:11:13.648[elastic_agent][warn] Possible transient error during checkin with fleet-server, retrying
14:11:13.648[elastic_agent][error] checkin retry loop was stopped
14:11:13.648[elastic_agent][warn] Possible transient error during checkin with fleet-server, retrying
14:11:13.648[elastic_agent][error] checkin retry loop was stopped
14:11:13.748[elastic_agent][info] Shutting down completed.
14:11:13.748[elastic_agent][info] Stats endpoint ([::]:6791) finished: accept tcp [::]:6791: use of closed network connection
14:11:13.748[elastic_agent][info] Shutting down completed.
14:11:13.748[elastic_agent][info] Stats endpoint ([::]:6791) finished: accept tcp [::]:6791: use of closed network connection
14:11:14.047[elastic_agent][info] APM instrumentation disabled
14:11:14.050[elastic_agent][info] Gathered system information
14:11:14.059[elastic_agent][info] Detected available inputs and outputs
14:11:14.059[elastic_agent][info] Capabilities file not found in /opt/Elastic/Agent/capabilities.yml
14:11:14.059[elastic_agent][info] Determined allowed capabilities
14:11:14.185[elastic_agent][info] Parsed configuration and determined agent is managed by Fleet
14:11:14.197[elastic_agent][info] Starting stats endpoint
14:11:14.197[elastic_agent][info] Metrics endpoint listening on: [::]:6791 (configured: http://:6791)
14:11:14.199[elastic_agent][info] Docker provider skipped, unable to connect: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
14:11:14.810[elastic_agent][info] restoring current policy from disk
14:11:14.819[elastic_agent][info] Source URI changed from "https://artifacts.elastic.co/downloads/" to "http://nexus.x.local/repository/proxy-raw-elasticagent/"
14:11:14.822[elastic_agent][info] Updating running component model
14:11:14.887[elastic_agent][info] Spawned new component system/metrics-default: Starting: spawned pid '22133'
14:11:14.887[elastic_agent][info] Spawned new unit system/metrics-default-system/metrics-system-aeee1beb-e6eb-4d9d-895e-8085d06d13c1: Starting: spawned pid '22133'
14:11:14.887[elastic_agent][info] Spawned new unit system/metrics-default: Starting: spawned pid '22133'
14:11:15.042[elastic_agent][info] Spawned new component fleet-server-default: Starting: spawned pid '22170'
14:11:15.042[elastic_agent][info] Spawned new unit fleet-server-default-fleet-server-fleet_server-aeddffe9-0d25-4571-8242-9f059f74ce0e: Starting: spawned pid '22170'
14:11:15.042[elastic_agent][info] Spawned new unit fleet-server-default: Starting: spawned pid '22170'
14:11:15.238[elastic_agent][info] Spawned new component log-default: Starting: spawned pid '22184'
14:11:15.238[elastic_agent][info] Spawned new unit log-default-logfile-system-aeee1beb-e6eb-4d9d-895e-8085d06d13c1: Starting: spawned pid '22184'
14:11:15.238[elastic_agent][info] Spawned new unit log-default: Starting: spawned pid '22184'
14:11:15.488[elastic_agent][info] Spawned new component filestream-monitoring: Starting: spawned pid '22225'
14:11:15.488[elastic_agent][info] Spawned new unit filestream-monitoring-filestream-monitoring-agent: Starting: spawned pid '22225'
14:11:15.488[elastic_agent][info] Spawned new unit filestream-monitoring: Starting: spawned pid '22225'
14:11:15.668[elastic_agent][info] Spawned new component beat/metrics-monitoring: Starting: spawned pid '22262'
14:11:15.668[elastic_agent][info] Spawned new unit beat/metrics-monitoring-metrics-monitoring-beats: Starting: spawned pid '22262'
14:11:15.668[elastic_agent][info] Spawned new unit beat/metrics-monitoring: Starting: spawned pid '22262'
14:11:15.921[elastic_agent][info] Spawned new component http/metrics-monitoring: Starting: spawned pid '22293'
14:11:15.922[elastic_agent][info] Spawned new unit http/metrics-monitoring: Starting: spawned pid '22293'
14:11:15.922[elastic_agent][info] Spawned new unit http/metrics-monitoring-metrics-monitoring-agent: Starting: spawned pid '22293'
14:11:15.922[elastic_agent][error] lazy acker: failed ack batch, enqueue for retry: [action_id: policy:fleet-server-policy:14:1, type: POLICY_CHANGE]
14:11:15.925[elastic_agent][info] Component state changed filestream-monitoring (STARTING->HEALTHY): Healthy: communicating with pid '22225'
14:11:15.925[elastic_agent][info] Component state changed fleet-server-default (STARTING->HEALTHY): Healthy: communicating with pid '22170'
14:11:15.925[elastic_agent][info] Component state changed system/metrics-default (STARTING->HEALTHY): Healthy: communicating with pid '22133'
14:11:15.925[elastic_agent][info] Component state changed log-default (STARTING->HEALTHY): Healthy: communicating with pid '22184'
14:11:15.925[elastic_agent][info] Component state changed beat/metrics-monitoring (STARTING->HEALTHY): Healthy: communicating with pid '22262'
14:11:16.029[elastic_agent][info] Unit state changed filestream-monitoring-filestream-monitoring-agent (STARTING->HEALTHY): Healthy
14:11:16.029[elastic_agent][info] Unit state changed filestream-monitoring (STARTING->HEALTHY): Healthy
14:11:16.031[elastic_agent][info] Unit state changed log-default-logfile-system-aeee1beb-e6eb-4d9d-895e-8085d06d13c1 (STARTING->HEALTHY): Healthy
14:11:16.031[elastic_agent][info] Unit state changed log-default (STARTING->HEALTHY): Healthy
14:11:16.041[elastic_agent][info] Unit state changed beat/metrics-monitoring-metrics-monitoring-beats (STARTING->HEALTHY): Healthy
14:11:16.041[elastic_agent][info] Unit state changed beat/metrics-monitoring (STARTING->HEALTHY): Healthy
14:11:16.054[elastic_agent][info] Unit state changed system/metrics-default-system/metrics-system-aeee1beb-e6eb-4d9d-895e-8085d06d13c1 (STARTING->HEALTHY): Healthy
14:11:16.054[elastic_agent][info] Unit state changed system/metrics-default (STARTING->HEALTHY): Healthy
14:11:16.062[elastic_agent][info] Unit state changed fleet-server-default-fleet-server-fleet_server-aeddffe9-0d25-4571-8242-9f059f74ce0e (STARTING->CONFIGURING): Re-configuring
14:11:16.063[elastic_agent][info] Unit state changed fleet-server-default (STARTING->CONFIGURING): Re-configuring
14:11:16.141[elastic_agent][info] Component state changed http/metrics-monitoring (STARTING->HEALTHY): Healthy: communicating with pid '22293'
14:11:16.232[elastic_agent][info] Unit state changed fleet-server-default-fleet-server-fleet_server-aeddffe9-0d25-4571-8242-9f059f74ce0e (CONFIGURING->HEALTHY): Running on policy with Fleet Server integration: fleet-server-policy
14:11:16.232[elastic_agent][info] Unit state changed fleet-server-default (CONFIGURING->HEALTHY): Running on policy with Fleet Server integration: fleet-server-policy
14:11:16.232[elastic_agent][info] Fleet gateway started
14:11:16.251[elastic_agent][info] Unit state changed http/metrics-monitoring-metrics-monitoring-agent (STARTING->HEALTHY): Healthy
14:11:16.251[elastic_agent][info] Unit state changed http/metrics-monitoring (STARTING->HEALTHY): Healthy
14:20:02.046[elastic_agent][info] signal "terminated" received
14:20:02.046[elastic_agent][info] Shutting down Elastic Agent and sending last events...
14:20:02.046[elastic_agent][warn] Possible transient error during checkin with fleet-server, retrying
14:20:02.046[elastic_agent][error] checkin retry loop was stopped
14:20:02.146[elastic_agent][info] Shutting down completed.
14:20:17.587[elastic_agent][info] APM instrumentation disabled
14:20:17.589[elastic_agent][info] Gathered system information
14:20:17.597[elastic_agent][info] Detected available inputs and outputs
14:20:17.597[elastic_agent][info] Capabilities file not found in /opt/Elastic/Agent/capabilities.yml
14:20:17.597[elastic_agent][info] Determined allowed capabilities
14:20:17.655[elastic_agent][info] Parsed configuration and determined agent is managed by Fleet
14:20:17.668[elastic_agent][info] Starting stats endpoint
14:20:17.668[elastic_agent][info] Metrics endpoint listening on: [::]:6791 (configured: http://:6791)
14:20:17.669[elastic_agent][info] Docker provider skipped, unable to connect: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
14:20:18.280[elastic_agent][info] restoring current policy from disk
14:20:18.290[elastic_agent][info] Source URI changed from "https://artifacts.elastic.co/downloads/" to "http://nexus.x.local/repository/proxy-raw-elasticagent/"
14:20:18.292[elastic_agent][info] Updating running component model
14:20:18.411[elastic_agent][info] Spawned new component fleet-server-default: Starting: spawned pid '22624'
14:20:18.411[elastic_agent][info] Spawned new unit fleet-server-default-fleet-server-fleet_server-aeddffe9-0d25-4571-8242-9f059f74ce0e: Starting: spawned pid '22624'
14:20:18.411[elastic_agent][info] Spawned new unit fleet-server-default: Starting: spawned pid '22624'
14:20:18.510[elastic_agent][info] Spawned new component log-default: Starting: spawned pid '22639'
14:20:18.510[elastic_agent][info] Spawned new unit log-default-logfile-system-aeee1beb-e6eb-4d9d-895e-8085d06d13c1: Starting: spawned pid '22639'
14:20:18.510[elastic_agent][info] Spawned new unit log-default: Starting: spawned pid '22639'
14:20:18.661[elastic_agent][info] Spawned new component system/metrics-default: Starting: spawned pid '22678'
14:20:18.661[elastic_agent][info] Spawned new unit system/metrics-default-system/metrics-system-aeee1beb-e6eb-4d9d-895e-8085d06d13c1: Starting: spawned pid '22678'
14:20:18.661[elastic_agent][info] Spawned new unit system/metrics-default: Starting: spawned pid '22678'
14:20:18.705[elastic_agent][info] Spawned new component filestream-monitoring: Starting: spawned pid '22685'
14:20:18.705[elastic_agent][info] Spawned new unit filestream-monitoring: Starting: spawned pid '22685'
14:20:18.705[elastic_agent][info] Spawned new unit filestream-monitoring-filestream-monitoring-agent: Starting: spawned pid '22685'
14:20:18.821[elastic_agent][info] Spawned new component beat/metrics-monitoring: Starting: spawned pid '22743'
14:20:18.821[elastic_agent][info] Spawned new unit beat/metrics-monitoring-metrics-monitoring-beats: Starting: spawned pid '22743'
14:20:18.821[elastic_agent][info] Spawned new unit beat/metrics-monitoring: Starting: spawned pid '22743'
14:20:18.900[elastic_agent][info] Spawned new component http/metrics-monitoring: Starting: spawned pid '22771'
14:20:18.901[elastic_agent][info] Spawned new unit http/metrics-monitoring-metrics-monitoring-agent: Starting: spawned pid '22771'
14:20:18.901[elastic_agent][error] lazy acker: failed ack batch, enqueue for retry: [action_id: policy:fleet-server-policy:14:1, type: POLICY_CHANGE]
14:20:18.901[elastic_agent][info] Spawned new unit http/metrics-monitoring: Starting: spawned pid '22771'
14:20:18.904[elastic_agent][info] Component state changed system/metrics-default (STARTING->HEALTHY): Healthy: communicating with pid '22678'
14:20:18.904[elastic_agent][info] Component state changed log-default (STARTING->HEALTHY): Healthy: communicating with pid '22639'
14:20:18.904[elastic_agent][info] Component state changed fleet-server-default (STARTING->HEALTHY): Healthy: communicating with pid '22624'
14:20:18.904[elastic_agent][info] Component state changed filestream-monitoring (STARTING->HEALTHY): Healthy: communicating with pid '22685'
14:20:18.924[elastic_agent][info] Component state changed beat/metrics-monitoring (STARTING->HEALTHY): Healthy: communicating with pid '22743'
14:20:19.009[elastic_agent][info] Unit state changed filestream-monitoring (STARTING->HEALTHY): Healthy
14:20:19.009[elastic_agent][info] Unit state changed filestream-monitoring-filestream-monitoring-agent (STARTING->HEALTHY): Healthy
14:20:19.011[elastic_agent][info] Unit state changed log-default-logfile-system-aeee1beb-e6eb-4d9d-895e-8085d06d13c1 (STARTING->HEALTHY): Healthy
14:20:19.011[elastic_agent][info] Unit state changed log-default (STARTING->HEALTHY): Healthy
14:20:19.029[elastic_agent][info] Unit state changed system/metrics-default-system/metrics-system-aeee1beb-e6eb-4d9d-895e-8085d06d13c1 (STARTING->HEALTHY): Healthy
14:20:19.029[elastic_agent][info] Unit state changed system/metrics-default (STARTING->HEALTHY): Healthy
14:20:19.037[elastic_agent][info] Unit state changed beat/metrics-monitoring-metrics-monitoring-beats (STARTING->HEALTHY): Healthy
14:20:19.037[elastic_agent][info] Unit state changed beat/metrics-monitoring (STARTING->HEALTHY): Healthy
14:20:19.084[elastic_agent][info] Component state changed http/metrics-monitoring (STARTING->HEALTHY): Healthy: communicating with pid '22771'
14:20:19.195[elastic_agent][info] Unit state changed http/metrics-monitoring (STARTING->HEALTHY): Healthy
14:20:19.195[elastic_agent][info] Unit state changed http/metrics-monitoring-metrics-monitoring-agent (STARTING->HEALTHY): Healthy
14:20:19.212[elastic_agent][info] Unit state changed fleet-server-default-fleet-server-fleet_server-aeddffe9-0d25-4571-8242-9f059f74ce0e (STARTING->HEALTHY): Running on policy with Fleet Server integration: fleet-server-policy
14:20:19.212[elastic_agent][info] Fleet gateway started
14:20:19.212[elastic_agent][info] Unit state changed fleet-server-default (STARTING->HEALTHY): Running on policy with Fleet Server integration: fleet-server-policy
14:20:19.424[elastic_agent][info] Unit state changed fleet-server-default-fleet-server-fleet_server-aeddffe9-0d25-4571-8242-9f059f74ce0e (HEALTHY->CONFIGURING): Re-configuring
14:20:19.424[elastic_agent][info] Unit state changed fleet-server-default (HEALTHY->CONFIGURING): Re-configuring
14:20:24.501[elastic_agent][info] Unit state changed fleet-server-default-fleet-server-fleet_server-aeddffe9-0d25-4571-8242-9f059f74ce0e (CONFIGURING->HEALTHY): Running on policy with Fleet Server integration: fleet-server-policy
14:20:24.501[elastic_agent][info] Unit state changed fleet-server-default (CONFIGURING->HEALTHY): Running on policy with Fleet Server integration: fleet-server-policy
14:20:24.501.fleet_server[elastic_agent.fleet_server][info] Running on policy with Fleet Server integration: fleet-server-policy
14:20:42.056.fleet_server[elastic_agent.fleet_server][info] Starting policy coordinator
14:20:42.056.fleet_server[elastic_agent.fleet_server][info] Starting policy coordinator
14:20:42.056.fleet_server[elastic_agent.fleet_server][info] Starting policy coordinator
14:20:42.056.fleet_server[elastic_agent.fleet_server][info] Starting policy coordinator
14:20:42.056.fleet_server[elastic_agent.fleet_server][info] Starting policy coordinator
14:20:42.056.fleet_server[elastic_agent.fleet_server][info] Starting policy coordinator
14:20:42.056.fleet_server[elastic_agent.fleet_server][info] Starting policy coordinator
14:20:43.210.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:20:53.871.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:20:57.895.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:20:59.813.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:21:01.333.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:21:03.619.fleet_server[elastic_agent.fleet_server][info] Starting policy coordinator
14:21:03.619.fleet_server[elastic_agent.fleet_server][info] Starting policy coordinator
14:21:25.184.fleet_server[elastic_agent.fleet_server][info] Starting policy coordinator
14:21:31.339.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:21:45.325.fleet_server[elastic_agent.fleet_server][info] applying new local metadata
14:21:45.325.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:22:05.765.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:22:22.731.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:22:36.410.fleet_server[elastic_agent.fleet_server][info] ApiKey fail authentication
14:22:36.410.fleet_server[elastic_agent.fleet_server][info] fail checkin
14:22:43.257.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:23:11.845.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:23:22.301.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:23:28.585.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:23:31.428.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:23:32.710.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:23:35.609.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:23:37.576.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:23:44.453.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:23:50.839.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:23:52.683.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:23:52.935.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:24:17.394.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:24:21.969.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:24:37.462.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:24:42.408.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:25:00.682.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:25:13.790.fleet_server[elastic_agent.fleet_server][info] ack event
14:25:15.221.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:25:21.962.fleet_server[elastic_agent.fleet_server][info] ApiKey fail authentication
14:25:21.962.fleet_server[elastic_agent.fleet_server][info] fail checkin
14:25:29.176.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:25:44.693.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:25:47.640.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:25:53.097.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:26:01.446.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:26:05.735.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:26:07.301.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:26:07.301.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:26:08.489.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:26:11.877.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:26:13.222.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:26:13.474.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:26:14.002.fleet_server[elastic_agent.fleet_server][info] applying new local metadata
14:26:14.002.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:26:15.488.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:26:24.018.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:26:25.825.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:26:27.769.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:26:29.176.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:26:42.218.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:27:04.982.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:27:07.200.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:27:15.509.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:27:22.545.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:27:22.545.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:27:23.049.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:27:30.780.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:27:35.096.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:27:41.441.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:27:41.694.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:27:47.886.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:29:58.071.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:30:09.965.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:30:32.283.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:30:34.606.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:30:36.573.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:30:43.724.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:31:07.780.fleet_server[elastic_agent.fleet_server][info] applying new local metadata
14:31:07.780.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:31:39.953.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:31:57.495.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:32:23.061.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:32:35.283.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:32:57.416.fleet_server[elastic_agent.fleet_server][info] applying new components data
14:33:03.684.fleet_server[elastic_agent.fleet_server][info] applying new components data
```
---
### Author:  [@matled](https://discuss.elastic.co/u/matled)
#### Post date: [February 17, 2023, 1:39pm UTC](https://discuss.elastic.co/t/elastic-agents-fail-to-upgrade-from-fleet-shows-updating-status-for-more-than-a-week/325426/9 "2023-02-17T13:39:10Z")
Log of an Elastic-Agent tried to upgrade from 8.6.1 to 8.6.2 and in stuck update state:
```auto
06:51:30.562[elastic_agent][info] Component state changed filestream-monitoring (HEALTHY->STOPPED): Suppressing FAILED state due to restart for '1468' exited with code '2'
06:51:30.562[elastic_agent][info] Unit state changed filestream-monitoring-filestream-monitoring-agent (HEALTHY->STOPPED): Suppressing FAILED state due to restart for '1468' exited with code '2'
06:51:30.562[elastic_agent][info] Unit state changed filestream-monitoring (HEALTHY->STOPPED): Suppressing FAILED state due to restart for '1468' exited with code '2'
06:51:31.989[elastic_agent][info] Spawned new component filestream-monitoring: Starting: spawned pid '96664'
06:51:31.989[elastic_agent][info] Spawned new unit filestream-monitoring-filestream-monitoring-agent: Starting: spawned pid '96664'
06:51:31.989[elastic_agent][info] Spawned new unit filestream-monitoring: Starting: spawned pid '96664'
06:51:32.171[elastic_agent][info] Component state changed filestream-monitoring (STARTING->HEALTHY): Healthy: communicating with pid '96664'
06:51:32.280[elastic_agent][info] Unit state changed filestream-monitoring (STARTING->HEALTHY): Healthy
06:51:32.280[elastic_agent][info] Unit state changed filestream-monitoring-filestream-monitoring-agent (STARTING->HEALTHY): Healthy
08:49:46.754[elastic_agent][info] Component state changed filestream-monitoring (HEALTHY->STOPPED): Suppressing FAILED state due to restart for '96664' exited with code '2'
08:49:46.754[elastic_agent][info] Unit state changed filestream-monitoring-filestream-monitoring-agent (HEALTHY->STOPPED): Suppressing FAILED state due to restart for '96664' exited with code '2'
08:49:46.754[elastic_agent][info] Unit state changed filestream-monitoring (HEALTHY->STOPPED): Suppressing FAILED state due to restart for '96664' exited with code '2'
08:49:47.755[elastic_agent][info] Spawned new component filestream-monitoring: Starting: spawned pid '122307'
08:49:47.755[elastic_agent][info] Spawned new unit filestream-monitoring: Starting: spawned pid '122307'
08:49:47.755[elastic_agent][info] Spawned new unit filestream-monitoring-filestream-monitoring-agent: Starting: spawned pid '122307'
08:49:47.948[elastic_agent][info] Component state changed filestream-monitoring (STARTING->HEALTHY): Healthy: communicating with pid '122307'
08:49:48.057[elastic_agent][info] Unit state changed filestream-monitoring (STARTING->HEALTHY): Healthy
08:49:48.058[elastic_agent][info] Unit state changed filestream-monitoring-filestream-monitoring-agent (STARTING->HEALTHY): Healthy
10:29:26.565[elastic_agent][info] Component state changed filestream-monitoring (HEALTHY->STOPPED): Suppressing FAILED state due to restart for '122307' exited with code '2'
10:29:26.565[elastic_agent][info] Unit state changed filestream-monitoring-filestream-monitoring-agent (HEALTHY->STOPPED): Suppressing FAILED state due to restart for '122307' exited with code '2'
10:29:26.565[elastic_agent][info] Unit state changed filestream-monitoring (HEALTHY->STOPPED): Suppressing FAILED state due to restart for '122307' exited with code '2'
10:29:27.566[elastic_agent][info] Spawned new component filestream-monitoring: Starting: spawned pid '143787'
10:29:27.566[elastic_agent][info] Spawned new unit filestream-monitoring-filestream-monitoring-agent: Starting: spawned pid '143787'
10:29:27.566[elastic_agent][info] Spawned new unit filestream-monitoring: Starting: spawned pid '143787'
10:29:27.745[elastic_agent][info] Component state changed filestream-monitoring (STARTING->HEALTHY): Healthy: communicating with pid '143787'
10:29:27.854[elastic_agent][info] Unit state changed filestream-monitoring-filestream-monitoring-agent (STARTING->HEALTHY): Healthy
10:29:27.854[elastic_agent][info] Unit state changed filestream-monitoring (STARTING->HEALTHY): Healthy
11:02:02.369[elastic_agent][info] Component state changed filestream-monitoring (HEALTHY->STOPPED): Suppressing FAILED state due to restart for '143787' exited with code '2'
11:02:02.370[elastic_agent][info] Unit state changed filestream-monitoring-filestream-monitoring-agent (HEALTHY->STOPPED): Suppressing FAILED state due to restart for '143787' exited with code '2'
11:02:02.370[elastic_agent][info] Unit state changed filestream-monitoring (HEALTHY->STOPPED): Suppressing FAILED state due to restart for '143787' exited with code '2'
11:02:03.370[elastic_agent][info] Spawned new component filestream-monitoring: Starting: spawned pid '150802'
11:02:03.370[elastic_agent][info] Spawned new unit filestream-monitoring-filestream-monitoring-agent: Starting: spawned pid '150802'
11:02:03.370[elastic_agent][info] Spawned new unit filestream-monitoring: Starting: spawned pid '150802'
11:02:03.595[elastic_agent][info] Component state changed filestream-monitoring (STARTING->HEALTHY): Healthy: communicating with pid '150802'
11:02:03.703[elastic_agent][info] Unit state changed filestream-monitoring (STARTING->HEALTHY): Healthy
11:02:03.703[elastic_agent][info] Unit state changed filestream-monitoring-filestream-monitoring-agent (STARTING->HEALTHY): Healthy
12:13:26.289[elastic_agent][info] Component state changed filestream-monitoring (HEALTHY->STOPPED): Suppressing FAILED state due to restart for '150802' exited with code '2'
12:13:26.289[elastic_agent][info] Unit state changed filestream-monitoring-filestream-monitoring-agent (HEALTHY->STOPPED): Suppressing FAILED state due to restart for '150802' exited with code '2'
12:13:26.289[elastic_agent][info] Unit state changed filestream-monitoring (HEALTHY->STOPPED): Suppressing FAILED state due to restart for '150802' exited with code '2'
12:13:27.290[elastic_agent][info] Spawned new component filestream-monitoring: Starting: spawned pid '166412'
12:13:27.290[elastic_agent][info] Spawned new unit filestream-monitoring: Starting: spawned pid '166412'
12:13:27.290[elastic_agent][info] Spawned new unit filestream-monitoring-filestream-monitoring-agent: Starting: spawned pid '166412'
12:13:27.382[elastic_agent][info] Component state changed filestream-monitoring (STARTING->HEALTHY): Healthy: communicating with pid '166412'
12:13:27.491[elastic_agent][info] Unit state changed filestream-monitoring-filestream-monitoring-agent (STARTING->HEALTHY): Healthy
12:13:27.491[elastic_agent][info] Unit state changed filestream-monitoring (STARTING->HEALTHY): Healthy
14:11:13.648[elastic_agent][warn] Possible transient error during checkin with fleet-server, retrying
14:17:45.165[elastic_agent][error] Checkin request to fleet-server succeeded after 1 failures
14:20:02.122[elastic_agent][warn] Possible transient error during checkin with fleet-server, retrying
14:25:50.294[elastic_agent][info] signal "terminated" received
14:25:50.294[elastic_agent][info] Shutting down Elastic Agent and sending last events...
14:25:50.294[elastic_agent][error] failed accept conn info connection: accept tcp 127.0.0.1:6788: use of closed network connection
14:25:50.294[elastic_agent][warn] Possible transient error during checkin with fleet-server, retrying
14:25:50.294[elastic_agent][error] checkin retry loop was stopped
14:25:50.395[elastic_agent][info] Shutting down completed.
14:25:50.395[elastic_agent][info] Stats endpoint (127.0.0.1:6791) finished: accept tcp 127.0.0.1:6791: use of closed network connection
14:25:51.209[elastic_agent][info] APM instrumentation disabled
14:25:51.211[elastic_agent][info] Gathered system information
14:25:51.221[elastic_agent][info] Detected available inputs and outputs
14:25:51.221[elastic_agent][info] Capabilities file not found in /opt/Elastic/Agent/capabilities.yml
14:25:51.221[elastic_agent][info] Determined allowed capabilities
14:25:51.259[elastic_agent][info] Parsed configuration and determined agent is managed by Fleet
14:25:51.274[elastic_agent][info] Starting stats endpoint
14:25:51.274[elastic_agent][info] Metrics endpoint listening on: 127.0.0.1:6791 (configured: http://localhost:6791)
14:25:51.274[elastic_agent][info] Docker provider skipped, unable to connect: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
14:25:51.485[elastic_agent][info] restoring current policy from disk
14:25:51.498[elastic_agent][info] Source URI changed from "https://artifacts.elastic.co/downloads/" to "http://nexus.x.local/repository/proxy-raw-elasticagent/"
14:25:51.502[elastic_agent][info] Updating running component model
14:25:51.573[elastic_agent][info] Spawned new component log-default: Starting: spawned pid '195728'
14:25:51.573[elastic_agent][info] Spawned new unit log-default: Starting: spawned pid '195728'
14:25:51.573[elastic_agent][info] Spawned new unit log-default-logfile-system-e7a753ee-cb91-43ad-baa0-dc46eac5f844: Starting: spawned pid '195728'
14:25:51.625[elastic_agent][info] Spawned new component udp-default: Starting: spawned pid '195752'
14:25:51.626[elastic_agent][info] Spawned new unit udp-default-udp-udp-b437a902-3d93-48f0-8024-98482437ec3f: Starting: spawned pid '195752'
14:25:51.626[elastic_agent][info] Spawned new unit udp-default-udp-udp-d463c3e6-d3a9-4c03-82a6-ba63851c21b8: Starting: spawned pid '195752'
14:25:51.626[elastic_agent][info] Spawned new unit udp-default-udp-udp-1fa4754e-1f50-4bc8-a2d8-a88e51c53706: Starting: spawned pid '195752'
14:25:51.626[elastic_agent][info] Spawned new unit udp-default-udp-udp-382e57fd-2a37-4f73-8dbd-7d2038ff2c29: Starting: spawned pid '195752'
14:25:51.626[elastic_agent][info] Spawned new unit udp-default-udp-udp-a06f2d81-5545-4f65-b544-55a553377a30: Starting: spawned pid '195752'
14:25:51.626[elastic_agent][info] Spawned new unit udp-default-udp-udp-1cec846b-577e-43ac-9b8c-9428396655da: Starting: spawned pid '195752'
14:25:51.626[elastic_agent][info] Spawned new unit udp-default-udp-udp-cdd5144f-ac7a-4e53-bfd3-e8ea901c202b: Starting: spawned pid '195752'
14:25:51.626[elastic_agent][info] Spawned new unit udp-default-udp-udp-bc297100-3632-43f1-a9af-3d231639da6d: Starting: spawned pid '195752'
14:25:51.626[elastic_agent][info] Spawned new unit udp-default: Starting: spawned pid '195752'
14:25:51.626[elastic_agent][info] Spawned new unit udp-default-udp-udp-c88effcf-6dd3-43fd-a121-8f0235ecc075: Starting: spawned pid '195752'
14:25:51.626[elastic_agent][info] Spawned new unit udp-default-udp-cisco_asa-880f0456-0308-4a94-88bb-9925cd2feef8: Starting: spawned pid '195752'
14:25:51.626[elastic_agent][info] Spawned new unit udp-default-udp-udp-8462ad5c-15ba-4b09-b1fa-f81d2668e83b: Starting: spawned pid '195752'
14:25:51.626[elastic_agent][info] Spawned new unit udp-default-udp-panw-032ef257-2d77-40c0-8e71-0070be23d5e0: Starting: spawned pid '195752'
14:25:51.626[elastic_agent][info] Spawned new unit udp-default-udp-Cisco ISE-4de3b518-19d6-4dc2-b436-72dbab631c8b: Starting: spawned pid '195752'
14:25:51.741[elastic_agent][info] Spawned new component netflow-default: Starting: spawned pid '195805'
14:25:51.741[elastic_agent][info] Spawned new unit netflow-default-netflow-netflow-0233a326-2634-473f-ab9b-6b6e40036fd9: Starting: spawned pid '195805'
14:25:51.741[elastic_agent][info] Spawned new unit netflow-default: Starting: spawned pid '195805'
14:25:51.785[elastic_agent][info] Spawned new component tcp-default: Starting: spawned pid '195822'
14:25:51.785[elastic_agent][info] Spawned new unit tcp-default-tcp-tcp-26611d2f-e89a-4cdc-81b3-3553c96ccf8d: Starting: spawned pid '195822'
14:25:51.785[elastic_agent][info] Spawned new unit tcp-default: Starting: spawned pid '195822'
14:25:51.864[elastic_agent][info] Spawned new component endpoint-default: Starting: endpoint service runtime
14:25:51.864[elastic_agent][info] Spawned new unit endpoint-default-c983b399-037b-4dcd-b6fc-71d439364660: Starting: endpoint service runtime
14:25:51.864[elastic_agent][info] Spawned new unit endpoint-default: Starting: endpoint service runtime
14:25:52.018[elastic_agent][info] Spawned new component filestream-monitoring: Starting: spawned pid '195876'
14:25:52.018[elastic_agent][info] Spawned new unit filestream-monitoring: Starting: spawned pid '195876'
14:25:52.018[elastic_agent][info] Spawned new unit filestream-monitoring-filestream-monitoring-agent: Starting: spawned pid '195876'
14:25:52.021[elastic_agent][info] Component state changed netflow-default (STARTING->HEALTHY): Healthy: communicating with pid '195805'
14:25:52.021[elastic_agent][info] Component state changed log-default (STARTING->HEALTHY): Healthy: communicating with pid '195728'
14:25:52.021[elastic_agent][info] Component state changed tcp-default (STARTING->HEALTHY): Healthy: communicating with pid '195822'
14:25:52.021[elastic_agent][info] Component state changed udp-default (STARTING->HEALTHY): Healthy: communicating with pid '195752'
14:25:52.028[elastic_agent][error] 2023-02-17 13:25:52: debug: ProcFile.cpp:1038 Found 1 cgroups for pid(195872)
14:25:52.028[elastic_agent][error] 2023-02-17 13:25:52: debug: ProcFile.cpp:1044 cgroup: id=0 type= path=/system.slice/elastic-agent.service
14:25:52.028[elastic_agent][error] 2023-02-17 13:25:52: info: MainPosix.cpp:341 Verifying existing installation
14:25:52.028[elastic_agent][error] 2023-02-17 13:25:52: info: InstallLib.cpp:616 Running [/opt/Elastic/Endpoint/elastic-endpoint] [version --log stdout]
14:25:52.029[elastic_agent][error] 2023-02-17 13:25:52: debug: Exec.cpp:187 ChildMonitor is pid 195880 and monitoring pids 195872 and 195878
14:25:52.125[elastic_agent][info] Unit state changed tcp-default (STARTING->HEALTHY): Healthy
14:25:52.125[elastic_agent][info] Unit state changed tcp-default-tcp-tcp-26611d2f-e89a-4cdc-81b3-3553c96ccf8d (STARTING->HEALTHY): Healthy
14:25:52.126[elastic_agent][info] Unit state changed netflow-default (STARTING->HEALTHY): Healthy
14:25:52.126[elastic_agent][info] Unit state changed netflow-default-netflow-netflow-0233a326-2634-473f-ab9b-6b6e40036fd9 (STARTING->HEALTHY): Healthy
14:25:52.128[elastic_agent][info] Unit state changed log-default-logfile-system-e7a753ee-cb91-43ad-baa0-dc46eac5f844 (STARTING->HEALTHY): Healthy
14:25:52.128[elastic_agent][info] Unit state changed log-default (STARTING->HEALTHY): Healthy
14:25:52.152[elastic_agent][info] Unit state changed udp-default-udp-udp-d463c3e6-d3a9-4c03-82a6-ba63851c21b8 (STARTING->HEALTHY): Healthy
14:25:52.152[elastic_agent][info] Unit state changed udp-default-udp-udp-8462ad5c-15ba-4b09-b1fa-f81d2668e83b (STARTING->HEALTHY): Healthy
14:25:52.152[elastic_agent][info] Unit state changed udp-default-udp-udp-a06f2d81-5545-4f65-b544-55a553377a30 (STARTING->HEALTHY): Healthy
14:25:52.152[elastic_agent][info] Unit state changed udp-default-udp-udp-382e57fd-2a37-4f73-8dbd-7d2038ff2c29 (STARTING->HEALTHY): Healthy
14:25:52.152[elastic_agent][info] Unit state changed udp-default (STARTING->HEALTHY): Healthy
14:25:52.152[elastic_agent][info] Unit state changed udp-default-udp-udp-c88effcf-6dd3-43fd-a121-8f0235ecc075 (STARTING->HEALTHY): Healthy
14:25:52.152[elastic_agent][info] Unit state changed udp-default-udp-udp-1fa4754e-1f50-4bc8-a2d8-a88e51c53706 (STARTING->HEALTHY): Healthy
14:25:52.152[elastic_agent][info] Unit state changed udp-default-udp-udp-bc297100-3632-43f1-a9af-3d231639da6d (STARTING->HEALTHY): Healthy
14:25:52.152[elastic_agent][info] Unit state changed udp-default-udp-Cisco ISE-4de3b518-19d6-4dc2-b436-72dbab631c8b (STARTING->HEALTHY): Healthy
14:25:52.152[elastic_agent][info] Unit state changed udp-default-udp-cisco_asa-880f0456-0308-4a94-88bb-9925cd2feef8 (STARTING->HEALTHY): Healthy
14:25:52.152[elastic_agent][info] Unit state changed udp-default-udp-udp-1cec846b-577e-43ac-9b8c-9428396655da (STARTING->HEALTHY): Healthy
14:25:52.152[elastic_agent][info] Unit state changed udp-default-udp-udp-cdd5144f-ac7a-4e53-bfd3-e8ea901c202b (STARTING->HEALTHY): Healthy
14:25:52.152[elastic_agent][info] Unit state changed udp-default-udp-udp-b437a902-3d93-48f0-8024-98482437ec3f (STARTING->HEALTHY): Healthy
14:25:52.152[elastic_agent][info] Unit state changed udp-default-udp-panw-032ef257-2d77-40c0-8e71-0070be23d5e0 (STARTING->HEALTHY): Healthy
14:25:52.166[elastic_agent][info] Component state changed filestream-monitoring (STARTING->HEALTHY): Healthy: communicating with pid '195876'
14:25:52.229[elastic_agent][error] 2023-02-17 13:25:52: info: InstallLib.cpp:656 Installed endpoint is expected version (version: 8.6.1, compiled: Thu Jan 19 23:00:00 2023, branch: 8.6, commit: 96e72414928d28f380d6a41b03aff24bfdcb816a)
14:25:52.277[elastic_agent][info] Unit state changed filestream-monitoring (STARTING->HEALTHY): Healthy
14:25:52.277[elastic_agent][info] Unit state changed filestream-monitoring-filestream-monitoring-agent (STARTING->HEALTHY): Healthy
14:25:52.414[elastic_agent][info] Fleet gateway started
14:25:52.419[elastic_agent][info] Updating running component model
14:26:20.920[elastic_agent][info] Component state changed endpoint-default (STARTING->HEALTHY): Healthy: communicating with endpoint service
14:26:20.920[elastic_agent][info] Unit state changed endpoint-default (STARTING->HEALTHY): Applied policy {c983b399-037b-4dcd-b6fc-71d439364660}
14:26:40.921[elastic_agent][info] Unit state changed endpoint-default-c983b399-037b-4dcd-b6fc-71d439364660 (STARTING->HEALTHY): Applied policy {c983b399-037b-4dcd-b6fc-71d439364660}
14:28:52.294[elastic_agent][info] Component state changed filestream-monitoring (HEALTHY->STOPPED): Suppressing FAILED state due to restart for '195876' exited with code '2'
14:28:52.294[elastic_agent][info] Unit state changed filestream-monitoring (HEALTHY->STOPPED): Suppressing FAILED state due to restart for '195876' exited with code '2'
14:28:52.294[elastic_agent][info] Unit state changed filestream-monitoring-filestream-monitoring-agent (HEALTHY->STOPPED): Suppressing FAILED state due to restart for '195876' exited with code '2'
14:28:53.295[elastic_agent][info] Spawned new component filestream-monitoring: Starting: spawned pid '196782'
14:28:53.295[elastic_agent][info] Spawned new unit filestream-monitoring-filestream-monitoring-agent: Starting: spawned pid '196782'
14:28:53.295[elastic_agent][info] Spawned new unit filestream-monitoring: Starting: spawned pid '196782'
14:28:53.472[elastic_agent][info] Component state changed filestream-monitoring (STARTING->HEALTHY): Healthy: communicating with pid '196782'
14:28:53.581[elastic_agent][info] Unit state changed filestream-monitoring-filestream-monitoring-agent (STARTING->HEALTHY): Healthy
14:28:53.581[elastic_agent][info] Unit state changed filestream-monitoring (STARTING->HEALTHY): Healthy
```
---
### Author:  [@slogger](https://discuss.elastic.co/u/slogger)
#### Post date: [February 20, 2023, 7:40am UTC](https://discuss.elastic.co/t/elastic-agents-fail-to-upgrade-from-fleet-shows-updating-status-for-more-than-a-week/325426/10 "2023-02-20T07:40:35Z")
Im also experiencing errors on all my agents after updating them to 8.6.1:
```auto
09:39:03.908
elastic_agent
[elastic_agent][error] Unit state changed filestream-default-filestream-container-logs-7a716136-d362-42b5-9128-dc63bbaf756b-kubernetes-1bac1134-5619-4bd9-938d-0c680ff58a9a.coredns (STARTING->FAILED): Failed: pid '448' exited with code '-1'
09:39:03.908
elastic_agent
[elastic_agent][error] Unit state changed filestream-default-filestream-container-logs-7a716136-d362-42b5-9128-dc63bbaf756b-kubernetes-b9006164-da1a-49cb-af67-239269670315.proxy (STARTING->FAILED): Failed: pid '448' exited with code '-1'
09:39:03.908
elastic_agent
[elastic_agent][error] Unit state changed filestream-default-filestream-container-logs-7a716136-d362-42b5-9128-dc63bbaf756b-kubernetes-01b23588-8a55-4fff-94db-6018d6a2d588.aws-vpc-cni-init (STARTING->FAILED): Failed: pid '448' exited with code '-1'
09:39:03.908
elastic_agent
[elastic_agent][error] Unit state changed filestream-default-filestream-container-logs-7a716136-d362-42b5-9128-dc63bbaf756b-kubernetes-b9006164-da1a-49cb-af67-239269670315.ingress-controller (STARTING->FAILED): Failed: pid '448' exited with code '-1'
09:39:03.908
elastic_agent
[elastic_agent][error] Unit state changed filestream-default-filestream-container-logs-7a716136-d362-42b5-9128-dc63bbaf756b-kubernetes-246587fa-86cb-4264-bba0-954e62d34c93.application-controller (STARTING->FAILED): Failed: pid '448' exited with code '-1'
09:39:03.909
elastic_agent
[elastic_agent][error] Unit state changed filestream-default-filestream-container-logs-7a716136-d362-42b5-9128-dc63bbaf756b-kubernetes-510fece8-fd6c-4d23-9b94-43fbd3749c97.coredns (STARTING->FAILED): Failed: pid '448' exited with code '-1'
09:39:04.809
elastic_agent.metricbeat
[elastic_agent.metricbeat][error] Error fetching data for metricset http.json: error making http request: Get "http://unix/stats": dial unix /usr/share/elastic-agent/state/data/tmp/filestream-default.sock: connect: connection refused
09:39:04.809
elastic_agent.metricbeat
[elastic_agent.metricbeat][error] Error fetching data for metricset http.json: error making http request: Get "http://unix/stats": dial unix /usr/share/elastic-agent/state/data/tmp/filestream-default.sock: connect: connection refused
09:39:05.088
elastic_agent.filebeat
[elastic_agent.filebeat][warn] Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.
09:39:05.096
elastic_agent.filebeat
[elastic_agent.filebeat][warn] Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning.
```
---
### Author:  [@DougR](https://discuss.elastic.co/u/DougR)
#### Post date: [February 20, 2023, 7:23pm UTC](https://discuss.elastic.co/t/elastic-agents-fail-to-upgrade-from-fleet-shows-updating-status-for-more-than-a-week/325426/11 "2023-02-20T19:23:35Z")
While it's not a solution, a workaround can be found in [this issue](https://github.com/elastic/fleet-server/issues/828) in github. It may or may not be related to [a second issue](https://github.com/elastic/elastic-agent/issues/760).
I have used the following API call, as noted in the first of the two posts, and Elastic Agent has successfully upgraded and returned to a healthy status on the hosts I've tested it on:
```bash
export KIBANA=https://
export ES_ADMIN_USER=
export ES_ADMIN_PASSWORD=
curl --request POST \
--url ${KIBANA}/api/fleet/agents/${id}/upgrade \
--user "${ES_ADMIN_USER}:${ES_ADMIN_PASSWORD}" \
--header 'Content-Type: application/json' \
--header 'kbn-xsrf: as' \
--data '{"version": "8.6.2", "force": true}'
```
---
### Author:  [@DougR](https://discuss.elastic.co/u/DougR)
#### Post date: [February 27, 2023, 3:01pm UTC](https://discuss.elastic.co/t/elastic-agents-fail-to-upgrade-from-fleet-shows-updating-status-for-more-than-a-week/325426/12 "2023-02-27T15:01:18Z")
I think there may actually be a regression in Fleet. I just looked at my Fleet logs today, and I moved several agents from one policy to another in Fleet nearly a week ago, and that activity is still showing as in progress, even though I can look at the new policy and see the agents in the new policy and can also see data being gathered from them.
---
### Author:  [@DougR](https://discuss.elastic.co/u/DougR)
#### Post date: [February 27, 2023, 3:27pm UTC](https://discuss.elastic.co/t/elastic-agents-fail-to-upgrade-from-fleet-shows-updating-status-for-more-than-a-week/325426/13 "2023-02-27T15:27:54Z")
I've opened a bug report.
>
>
> \# Issue
>
> Since upgrading from Elastic Cloud 8.5.3, I have been unable to manag…e Elastic Agent integrations through Fleet in Kibana.
>
> I initially upgraded from 8.5.3-\>8.6.1 and scheduled an Agent upgrade. A few of the agents upgraded as expected, the remaining remained stuck in "Updating..." status in Fleet for more than a week. I later updated to 8.6.2 and continued to experience the same issue.
>
> I cancelled out the "Updating..." status per the instructions \[in this post\](https://discuss.elastic.co/t/fleet-server-status-stuck-at-updating-but-seems-to-be-working-normally/318193/2), which returned the agents to Healthy status, as promised. A second attempt at upgrading yielded the same results. After cancelling out the "Updating..." status again, I determined that I was able to schedule small numbers of agents to upgrade immediately and it would usually succeed, as long as they were in "Healthy" status, but if an "Offline" or even a "Not Healthy" agent was included, the upgrade would fail. If I attempted to schedule the update for a later time, or if I included too many agents at once (usually around 7), the upgrade would fail.
>
> Additionally, I observed this morning when I checked the "Agent Activity" log in fleet that multiple activities showing "Reassigning X of Y agents to \[new policy\]" that I executed approximately a week ago are still showing as in progress. However, in this case, when I investigate, all agents have been reassigned to the new policy and are reporting in as expected. I don't know whether the two items are related.
>
> \# Deployment
>
> My deployment is:
>
> \* Elastic Cloud 8.6.2
> \* Fleet-Managed Elastic Agent in both Windows and Linux
> \* Discussion Forum: https://discuss.elastic.co/t/elastic-agents-fail-to-upgrade-from-fleet-shows-updating-status-for-more-than-a-week/325426/12
>
> \# Steps to Reproduce
>
> 1. Upgrade Elastic Cloud to 8.6.1 or 8.6.2
> 2. Schedule an agent upgrade in Fleet.
---
### Author:  [@Julia\_Bardi](https://discuss.elastic.co/u/Julia_Bardi)
#### Post date: [February 28, 2023, 4:57pm UTC](https://discuss.elastic.co/t/elastic-agents-fail-to-upgrade-from-fleet-shows-updating-status-for-more-than-a-week/325426/14 "2023-02-28T16:57:37Z")
Thanks for opening the bug report, this indeed seems like a regression in recent versions, we can reproduce it locally. Please check the latest status on the github issue, hopefully we can find a fix soon.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 28, 2023, 4:58pm UTC](https://discuss.elastic.co/t/elastic-agents-fail-to-upgrade-from-fleet-shows-updating-status-for-more-than-a-week/325426/15 "2023-03-28T16:58:15Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.