# Update to 8.3.1 from 8.3.0 has broken Fleet - please help!

**URL:** https://discuss.elastic.co/t/update-to-8-3-1-from-8-3-0-has-broken-fleet-please-help/308654
**Category:** Elastic Security
**Created:** [July 1, 2022, 11:40am UTC](https://discuss.elastic.co/t/update-to-8-3-1-from-8-3-0-has-broken-fleet-please-help/308654 "2022-07-01T11:40:14Z")
**Posts on this page:** 12
**Page:** 1

<div class="post-metadata">

### Author: ![finbarr996](https://avatars.discourse-cdn.com/v4/letter/f/db5fbb/32.png) [@finbarr996](https://discuss.elastic.co/u/finbarr996)
#### Post date: [July 1, 2022, 11:40am UTC](https://discuss.elastic.co/t/update-to-8-3-1-from-8-3-0-has-broken-fleet-please-help/308654/1 "2022-07-01T11:40:14Z")

</div>

Upgraded to 8.3 a few days ago - everything working fine.  
Updated to 8.3.1 this morning and now the elastic-agent service won't start on my single on-prem node - this is also the Fleet Server.

Error in the logs is:  
`Error: could not read overwrites: fail to read configuration /etc/elastic-agent/fleet.enc for the elastic-agent: fail to decode bytes: cipher: message authentication failed`

I've not changed anything in the config files at all prior to the update.  
I stopped all the elastic services on the host, (elasticsearch, kibana, logstash, metricbeat, filebeat, elastic-agent) ran the update and restarted the elastic services again.  
Everything restarted except the 'elastic-agent' service, which fails with the error described above.

---

<div class="post-metadata">

### Author: ![somerandomitperson](https://avatars.discourse-cdn.com/v4/letter/s/919ad9/32.png) [@somerandomitperson](https://discuss.elastic.co/u/somerandomitperson)
#### Post date: [July 3, 2022, 1:52pm UTC](https://discuss.elastic.co/t/update-to-8-3-1-from-8-3-0-has-broken-fleet-please-help/308654/2 "2022-07-03T13:52:34Z")

</div>

I'm having the same issue as well, and I think that I made it worse. I tried recreating the fleet.enc and checking permissions on it, and I think I broke it worse. Good thing I took a snapshot.

---

<div class="post-metadata">

### Author: ![somerandomitperson](https://avatars.discourse-cdn.com/v4/letter/s/919ad9/32.png) [@somerandomitperson](https://discuss.elastic.co/u/somerandomitperson)
#### Post date: [July 3, 2022, 1:57pm UTC](https://discuss.elastic.co/t/update-to-8-3-1-from-8-3-0-has-broken-fleet-please-help/308654/3 "2022-07-03T13:57:09Z")

</div>

Update: Downgraded to 8.3.0 for the agent only with  
apt install elastic-agent=8.3.0 and my fleet server is working again. @finbarr996

That may get you working until Elastic can fix this.

---

<div class="post-metadata">

### Author: ![finbarr996](https://avatars.discourse-cdn.com/v4/letter/f/db5fbb/32.png) [@finbarr996](https://discuss.elastic.co/u/finbarr996)
#### Post date: [July 3, 2022, 11:07pm UTC](https://discuss.elastic.co/t/update-to-8-3-1-from-8-3-0-has-broken-fleet-please-help/308654/4 "2022-07-03T23:07:14Z")

</div>

Hi Nathan,  
Thank you for the tip - that worked perfectly - everything is back up and running again!  
I wasn't aware of the ability to use apt to downgrade to a previous version - so thank you for that!

Hopefully Elastic will fix this soon.

Cheers,  
John.

---

<div class="post-metadata">

### Author: ![aleksmaus](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aleksmaus/32/56530_2.png) [@aleksmaus](https://discuss.elastic.co/u/aleksmaus)
#### Post date: [July 6, 2022, 6:12pm UTC](https://discuss.elastic.co/t/update-to-8-3-1-from-8-3-0-has-broken-fleet-please-help/308654/5 "2022-07-06T18:12:18Z")

</div>

@finbarr996 How did you run the upgrade? Could you provide the steps?  
Trying to get some repro case here.  
Just tested the upgrade via fleet (initiated from Kibana) from 8.3.0 to 8.3.1 and it worked fine.

---

<div class="post-metadata">

### Author: ![somerandomitperson](https://avatars.discourse-cdn.com/v4/letter/s/919ad9/32.png) [@somerandomitperson](https://discuss.elastic.co/u/somerandomitperson)
#### Post date: [July 6, 2022, 6:31pm UTC](https://discuss.elastic.co/t/update-to-8-3-1-from-8-3-0-has-broken-fleet-please-help/308654/6 "2022-07-06T18:31:36Z")

</div>

To upgrade the initial agent, the fleet server, I did apt update and apt upgrade.

After that, the elastic-agent.service was not able to start, so I could not start my fleet server. I got the same error that John got in his initial message.

---

<div class="post-metadata">

### Author: ![finbarr996](https://avatars.discourse-cdn.com/v4/letter/f/db5fbb/32.png) [@finbarr996](https://discuss.elastic.co/u/finbarr996)
#### Post date: [July 7, 2022, 1:21pm UTC](https://discuss.elastic.co/t/update-to-8-3-1-from-8-3-0-has-broken-fleet-please-help/308654/7 "2022-07-07T13:21:45Z")

</div>

I upgraded my single on-prem Elastic node by stopping all the Elastic services, and then followed with apt udate, apt upgrade, so Elasticsearch, Kibana, Logstash, Metricbeat and elastic-agent were all upgraded from 8.3 to 8.3.1. I then rebooted the box and started the services.

Elastic-agent was the only service that wouldn't start - it seems to me that for whatever reason that elastic-agent was no longer able to decrypt the fleet.enc file and read the configuration items from there, so it failed to start.

I hope this helps.

---

<div class="post-metadata">

### Author: ![PublicName](https://avatars.discourse-cdn.com/v4/letter/p/74df32/32.png) [@PublicName](https://discuss.elastic.co/u/PublicName)
#### Post date: [July 7, 2022, 7:17pm UTC](https://discuss.elastic.co/t/update-to-8-3-1-from-8-3-0-has-broken-fleet-please-help/308654/8 "2022-07-07T19:17:45Z")

</div>

Confirmed 8.3.1 fails for fleet updates. Host is CentOS 8 Stream and also happens on Oracle Linux 8.3.

I can't recall the error message as it was several days ago but it was about the payload size being outside the expected. This was an error awhile ago.

---

<div class="post-metadata">

### Author: ![finbarr996](https://avatars.discourse-cdn.com/v4/letter/f/db5fbb/32.png) [@finbarr996](https://discuss.elastic.co/u/finbarr996)
#### Post date: [July 8, 2022, 3:52pm UTC](https://discuss.elastic.co/t/update-to-8-3-1-from-8-3-0-has-broken-fleet-please-help/308654/9 "2022-07-08T15:52:38Z")

</div>

Having rolled back to 8.3 which works perfectly, I just upgraded to 8.3.2 and the elastic-agent service fails to start with exactly the same error as my original post. ☹

I'll be rolling back to 8.3 again I guess.

---

<div class="post-metadata">

### Author: ![rbos](https://avatars.discourse-cdn.com/v4/letter/r/b3f665/32.png) [@rbos](https://discuss.elastic.co/u/rbos)
#### Post date: [July 19, 2022, 4:03pm UTC](https://discuss.elastic.co/t/update-to-8-3-1-from-8-3-0-has-broken-fleet-please-help/308654/10 "2022-07-19T16:03:18Z")

</div>

I experienced this error going from 8.3.1 to 8.3.2 (RHEL7, using the package repos for elastic-agent). Downgrading back to 8.3.1 fixed it.

---

<div class="post-metadata">

### Author: ![VamPikmin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vampikmin/32/22367_2.png) [@VamPikmin](https://discuss.elastic.co/u/VamPikmin)
#### Post date: [July 20, 2022, 8:32am UTC](https://discuss.elastic.co/t/update-to-8-3-1-from-8-3-0-has-broken-fleet-please-help/308654/11 "2022-07-20T08:32:37Z")

</div>

I'm finding the same thing on Ubuntu 20.04 (8.3.2)

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 17, 2022, 8:33am UTC](https://discuss.elastic.co/t/update-to-8-3-1-from-8-3-0-has-broken-fleet-please-help/308654/12 "2022-08-17T08:33:04Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
