Kibana errors after changing encryptionKey - Failed to decrypt "apiKey" attribute: Unsupported state or unable to authenticate data

[PodarcisMuralis (Murat K.)]

I use elasticstack 8.5.3 and have 2 Logstash, 5 ELS and 1 Kibana nodes.

I was cleaning the older kibana system indices ( upgraded from 7.17.7) and deleted .security_7 index also and had to create all built in users again.

I had to also create new encryption key for kibana by using this :
kibana-encryption-keys

xpack.security.encryptionKey: "xxx"
xpack.encryptedSavedObjects.encryptionKey: "xxx"
xpack.reporting.encryptionKey: "xxx"

But after that I keep getting this error:

Failed to decrypt "apiKey" attribute: Unsupported state or unable to authenticate data

or specifically

Error: Unable to decrypt attribute "apiKey"
    at loadRule (/../kibana-8.5.3/x-pack/plugins/alerting/server/task_runner/rule_loader.js:43:11)
    at TaskRunner.prepareToRun (/../kibana-8.5.3/x-pack/plugins/alerting/server/task_runner/task_runner.js:494:12)
    at TaskRunnerTimer.runWithTimer (/../kibana-8.5.3/x-pack/plugins/alerting/server/task_runner/task_runner_timer.js:57:20)
    at TaskRunner.run (/../kibana-8.5.3/x-pack/plugins/alerting/server/task_runner/task_runner.js:611:30)
    at TaskManagerRunner.run (/../kibana-8.5.3/x-pack/plugins/task_manager/server/task_running/task_runner.js:299:22)

It is flooding my logs.

I read this documentation: xpack-security-secure-saved-objects

If I try to use just

xpack.encryptedSavedObjects:
  encryptionKey: "newKey"
  keyRotation:
    decryptionOnlyKeys: ["oldKey"]

I get this error:

Re-authentication can not be handled

I deleted all saved objects on kibana.
I do not have any rules, alers etc.
It is just empty kibana but still get these errors.
I also deleted old version indices
.kibana_7
.kibana_task_manager_7
.kibana_event_log_7
but still get this error.

Is this is a kibana bug as stated here?
"unable to decrypt" error

[PodarcisMuralis (Murat K.)]

After spending a day on this problem I found out the issue.

I have 2 Kibana Spaces: Default and Custom.
I was checking the rules and alerts wrongly on Custom.
But I had many rules on Default Spaces.
So I changed to Default Space.
Stack Management - Rules and Connectors - > Rules
Click on the 3 dots on the right side of the rules.
Click on the Update API Key.
It worked like a magic.

I have these settings in kibana.yml

xpack.security.encryptionKey: "newKey"
xpack.encryptedSavedObjects.encryptionKey: "newKey"
xpack.reporting.encryptionKey: "newKey"