Kibana Security Exception When Viewing Rules/Alerts

[DougR]

I'm configuring Elastic Security with for Elastic Stack 7.16.3. When I visit Stack Management->Rules and Connectors in Kibana, I see the following:

...and this is the specific error that shows up when I hover over the Error status:

When I look at the actual logs, it's littered with log entries like this:

{
    "type": "log",
    "@timestamp": "2022-02-14T17:19:43+00:00",
    "tags": [
        "error",
        "plugins",
        "alerting"
    ],
    "pid": 8,
    "message": "Executing Alert default:monitoring_alert_cpu_usage:b32b3780-22bd-11ec-9456-c3f758019dc7 has resulted in Error: security_exception: [security_exception] Reason: missing authentication credentials for REST request [/_security/user/_has_privileges], caused by: \"\""
}

Here's the relevant portion of my kibana.yml:

# Configure connection to Elasticsearch
elasticsearch:
  username: '${KIBANA_SYSTEM_USERNAME}'
  password: '${KIBANA_SYSTEM_PASSWORD}'
  ssl:
    alwaysPresentCertificate: false
    certificate: '/usr/share/kibana/config/certs/cert.pem'
    key: '/usr/share/kibana/config/certs/key.pem'
    certificateAuthorities:
      - '/usr/share/kibana/config/certs/ca_bundle.pem'
    verificationMode: 'none'

Other than this, Kibana appears to connect just fine with this configuration (i.e., I can perform admin functions and search Elasticsearch from Kibana).

I previously had xpack.security.disabled: false set on this stack, if it matters, and so I was wondering whether this reply to Issue #118520 might apply.

I don't currently have any credentials, etc. configured, so is there an index that I can delete/recreate at restart that will resolve this issue, or some similar solution? I have data, which I need to preserve, so simply deleting my stack and restarting from scratch isn't really an option.

[DougR]

Figured this out this morning. According to the Authorization section of the Alerting Setup page:

Rules are authorized using an API key associated with the last user to edit the rule. This API key captures a snapshot of the user’s privileges at the time of edit and is subsequently used to run all background tasks associated with the rule, including condition checks, like Elasticsearch queries, and action executions. The following rule actions will re-generate the API key:

• Creating a rule
• Enabling a disabled rule
• Updating a rule

Since these rules were for a brand new stack, they'd never been saved before by a user. Once I went in and saved the rules with the elastic user, the rules started running as expected. Again, I had initially configured this stack with xpack.security.enabled: false, so this may not be an issue if security is configured initially.