Rule execution failure: Rule registry writing is disabled

I've created a rule for logs threshold, and there is an error while running the rule:
'Process check wildfly' (that's the name of the rule) - Rule registry writing is disabled due to an error during Rule Data Client Initialization.

A sample log from Kibana:

[2022-09-20T10:33:25.671+02:00][ERROR][plugins.alerting] RuleDataWriteDisabledError: Rule registry writing is disabled due to an error during Rule Data Client initialization.
    at RuleDataClient.initializeWriter (/srv/int/kibana-8.4.0/x-pack/plugins/rule_registry/server/rule_data_client/rule_data_client.js:173:13)
    at RuleDataClient.getWriter (/srv/int/kibana-8.4.0/x-pack/plugins/rule_registry/server/rule_data_client/rule_data_client.js:147:45)
    at Object.executor (/srv/int/kibana-8.4.0/x-pack/plugins/rule_registry/server/utils/create_lifecycle_executor.js:59:53)
    at /srv/int/kibana-8.4.0/x-pack/plugins/alerting/server/task_runner/task_runner.js:259:30
    at AsyncLocalStorage.run (node:async_hooks:327:14)
    at ExecutionContextService.withContext (/srv/int/kibana-8.4.0/node_modules/@kbn/core-execution-context-server-internal/target_node/execution_context_service.js:91:30)
    at TaskRunner.executeRule (/srv/int/kibana-8.4.0/x-pack/plugins/alerting/server/task_runner/task_runner.js:256:66)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at promiseResult (/srv/int/kibana-8.4.0/x-pack/plugins/alerting/server/lib/result_type.js:44:17)
    at TaskRunner.loadRuleAttributesAndRun (/srv/int/kibana-8.4.0/x-pack/plugins/alerting/server/task_runner/task_runner.js:453:25)
    at errorAsRuleTaskRunResult (/srv/int/kibana-8.4.0/x-pack/plugins/alerting/server/task_runner/task_runner.js:837:12)
    at TaskRunner.run (/srv/int/kibana-8.4.0/x-pack/plugins/alerting/server/task_runner/task_runner.js:515:9)
    at TaskManagerRunner.run (/srv/int/kibana-8.4.0/x-pack/plugins/task_manager/server/task_running/task_runner.js:299:22)
[2022-09-20T10:33:25.712+02:00][ERROR][plugins.alerting] Executing Rule default:logs.alert.document.count:8bee1930-38b0-11ed-8e7d-390642998641 has resulted in Error: Rule registry writing is disabled due to an error during Rule Data Client initialization.

Can anyone help with this?

Hi @alytkowski,

RuleDataClient initialization can fail for a few different reasons. In order to determine what's going wrong here, some additional information would be helpful.

  1. Have any changes been made to the ILM policy ".alerts-ilm-policy"? The rule data client initialization can fail if the ILM policy has been edited to roll alerts indices over to cold or snapshot tiers. ([Response Ops] RuleDataClient initialization fails if any alerts indices are snapshots · Issue #139969 · elastic/kibana · GitHub)
  2. We've also seen RuleDataClient initialization fail if Elasticsearch is overloaded when the RuleDataClient attempts to initialize.

In either case, the Kibana logs that show the initialization failure would be helpful to identify the cause. You can filter for the logs for plugins.ruleRegistry to narrow them down when searching for the initialization failure. Also, for security rules, the RuleDataClient will attempt to initialize the first time a rule tries to write an alert after Kibana starts up.