[Alerting] My alerts are broken after upgrading Elastic Cloud to 7.10.2

[nickbabkin]

Hi!
After upgrading ES cluster and Kibana to 7.10.2, I noticed that some of my alerts have started to throw error and hence refuse to work due to these error:

An error occurred when decrypting the alert.
Unable to decrypt attribute "apiKey"

I also get the same error every time I create a new alert.

My assumption is that it can happen when xpack.encryptedSavedObjects parameter is not set in kibana (or a different key than was used before is set hence rendering decryption impossible). However, I don't have any control over this parameter in Elastic Cloud and I never changed/deleted it by myself.

Requesting community / dev help with this. Maybe there's a way to recreate alerts index and encrypt it with the current key? I tried to delete/create API keys, disable and reenable alerts - none of this helped. Also, how can I ensure same issue doesn't appear next time when I upgrade?

Screenshots attached.

Thanks!

Screenshot 2021-02-10 at 5.16.18 PM|510x200
Screenshot 2021-02-10 at 5.15.58 PM|690x453

[warkolm (Mark Walkom)]

Have you reached out to the Support team about this one?

[nickbabkin]

Hi!
Not yet, mostly due to the fact that I'm usually getting help from the community way faster than from the support

[stephenb (Stephen Brown)]

@nickbabkin

I had a potentially similar issues I was told but did not do detailed analysis there is a potential race condition bug that causes that issue especially perhaps when Kibana under load.

Try disabling all those alerts, then open an alert, make a trivial change add a space, take out a space and save it and re-enable.

See if that works not a great suggestion but try one let us know... I would still persue support