After spending a day on this problem I found out the issue.
I have 2 Kibana Spaces: Default and Custom.
I was checking the rules and alerts wrongly on Custom.
But I had many rules on Default Spaces.
So I changed to Default Space.
Stack Management - Rules and Connectors - > Rules
Click on the 3 dots on the right side of the rules.
Click on the Update API Key.
It worked like a magic.
I have these settings in kibana.yml
xpack.security.encryptionKey: "newKey"
xpack.encryptedSavedObjects.encryptionKey: "newKey"
xpack.reporting.encryptionKey: "newKey"