Built in alerts fail to run

[radovan]

Hi,

I decided to create a monitoring cluster and after a few problems at the start all is running now. There are metricbeat clients on the elastic nodes with only elastic/logstash/kibana modules enabled, logs are flowing onto the monitoring cluster to .monitoring-XXX indexes, but built in monitoring rules fail to run. Do I need to do some initialization first or something I need to setup? All of the 14 alerts says only this, when I open them:

An error occurred when running the rule.
illegal_argument_exception, caused by: ""

The version of production and monitoring elastic stack is 7.13.3. Monitoring cluster has 2 nodes, first one is master and second one is ingest/data node.

[gbeltramelli]

same problem!

[monfera (Robert Monfera)]

This may be a problem in Kibana Alerting rules, as by default, it enables all rules associated with remote clusters used in CCR/CCS. If it's the case, all alerting rules created manually should continue to work fine.

Here's a possible fix/workaround:

For 7.7+, add the following in kibana.yml of the "monitoring" cluster:

monitoring.ui.ccs.enabled: false

or add the remote_cluster_client role to all of their nodes.

[radovan]

I did both, first the kibana setting and after restarting kibana, these 4 remained in error state with same error message:

Then I added remote_cluster_client role to my 2 nodes that are part of the monitoring cluster, restarted them but the errors still occur.

[monfera (Robert Monfera)]

It's a bug to begin with, and apparently the workarounds didn't work out. We use github for bug reporting. Would you mind putting your specific details into a comment, if you feel it's appropriate for [Monitoring] `remote_cluster_client` role shouldn't be required to use Monitoring · Issue #93432 · elastic/kibana · GitHub or if not, could you please file a new ticket and link here too?

[radovan]

Don't know what happened, I just let it be during the weekend and now even the last 4 alerts seem they have no problem now. So my issue is probably fixed, thank you!