Fleet initial error

Hi
after upgrade from 8.9.1 to 8.11.1 , I have error on fleet page "unable to initialize fleet, uninstall token is missing the token"

1 Like

Hi @mohd_sa im also facing same issue, any luck?

i did not find any solution

so whtas the next move? just asking. We got this error because kibana instance got replaced as a part of autoscaling group health check and it resulted with the error uninstall token missing when clicked on fleet tab in kibana interface. sad part is elk documentation also doesnt have any clue of this.

Hi folks, We've seen this reported in a few places so far on the 8.11.1 release and we're taking a look. Once we have a public issue or know a little bit more about the root cause I'll post an update here.

One thing that'd be helpful to know: does anyone encountering this issue make use of Kibana's saved objects encryption feature? The errors message we're seeing here is related to saved object encryption, and we're trying to track down the root cause.

Many thanks, and I hope to have an update for you all soon.

Hello
All my rules (under Alerts and Insights) were generating api key errors
I disabled them and did Update Api Key for all of them.
After that I did not have any api key errors anymore.

But fleet is still is still giving

My environment is on Openshift
The kibana.yml is stored in a secret and has

xpack:
encryptedSavedObjects:
encryptionKey: ................

1 Like

We just discovered that we do have issues with Rules:

Thanks @HansPeterSloot - this is very helpful as it appears to confirm this is a broader issue with encrypted saved objects that isn't limited to Fleet. We've got a few teams looking at this internally today.

@kpollich is there any public issue on github about this?

We were planning to upgrade to 8.11, but after seeing that this can cause issues with Alert Rules, which is our main use case we can not upgrade anymore and risk any impacts to our operations.

Would like to follow some issue to know when it is safe to Upgrade.

@leandrojmp - I've just created [Encrypted Saved Objects] Multiple plugins experiencing errors with encrypted saved objects in 8.11.1 · Issue #171630 · elastic/kibana · GitHub to track this publicly. I'll keep this updated with our findings as we work, so please feel free to follow along there :pray:

1 Like

Question for folks here: is anyone using Elastic Cloud on Kubernetes (Overview | Elastic Cloud on Kubernetes [2.10] | Elastic), or are these all self-hosted clusters?

We've seen this reported in support cases internally on ECK, and I see Hans's example above is on OpenShift (so similar K8's concerns), and I'm trying to determine whether these are related.

Many thanks. We have 3-4 engineering teams all looking at this urgently today and I will continue to provide updates here and on the public issue.

If we want to proceed to normal operations isnt reverting back (from a restore) to 8.10 a possible way forward until this is resolved. As it does not look like this is an easy problem (wrong encryption key in a new software release?)

It's not clear to me whether rolling back to a previous version would fix the issue, but it's certainly worth a try. I'd be curious to know if you're running on ECK, and if you could provide any Kibana logs from before and after the upgrade.

I've posted a lot of findings and details over on [Encrypted Saved Objects] Multiple plugins experiencing errors with encrypted saved objects in 8.11.1 · Issue #171630 · elastic/kibana · GitHub today, for anyone following along.

@HansPeterSloot could you provide these ?

We are running on Openshift.
I do not have any logs from before.

1 Like

I am now restoring a snapshot from before the upgrade on a separate cluster.

Hi all, I've posted a workaround to unblock Fleet in the instance of this error here: [Encrypted Saved Objects] Multiple plugins experiencing errors with encrypted saved objects in 8.11.1 · Issue #171630 · elastic/kibana · GitHub.

We've validated this workaround internally and in production for a few users, but please let me know if you run into any issues with it.

As far as a root cause goes, I'm afraid I don't have much news there. We're still working with a few teams at trying to track down the root cause here and why this happened specifically during the 8.11 release cycle. I will post an update as soon as I have one.

Thanks for your patience :pray:

im using self-hosted fleet in aws ec2. im facing below errors

  1. im getting unable to intilaise fleet error in an intermittent way, means after around two minutes im logged out from my account and when i re-login i can see the fleet option back again.
  2. im unable to download the policy or view the policy which is created while adding a fleet server. as a result of this when i run the commands to add fleet server the response im getting is waiting for fleet-server policy.

can you help on this? please. i have attached the screenshot.

Thanks.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.