Built in alerts fail to run


I decided to create a monitoring cluster and after a few problems at the start all is running now. There are metricbeat clients on the elastic nodes with only elastic/logstash/kibana modules enabled, logs are flowing onto the monitoring cluster to .monitoring-XXX indexes, but built in monitoring rules fail to run. Do I need to do some initialization first or something I need to setup? All of the 14 alerts says only this, when I open them:

An error occurred when running the rule.
illegal_argument_exception, caused by: ""

The version of production and monitoring elastic stack is 7.13.3. Monitoring cluster has 2 nodes, first one is master and second one is ingest/data node.

same problem!

This may be a problem in Kibana Alerting rules, as by default, it enables all rules associated with remote clusters used in CCR/CCS. If it's the case, all alerting rules created manually should continue to work fine.

Here's a possible fix/workaround:

For 7.7+, add the following in kibana.yml of the "monitoring" cluster:

monitoring.ui.ccs.enabled: false

or add the remote_cluster_client role to all of their nodes.

I did both, first the kibana setting and after restarting kibana, these 4 remained in error state with same error message:

Then I added remote_cluster_client role to my 2 nodes that are part of the monitoring cluster, restarted them but the errors still occur.

It's a bug to begin with, and apparently the workarounds didn't work out. We use github for bug reporting. Would you mind putting your specific details into a comment, if you feel it's appropriate for [Monitoring] `remote_cluster_client` role shouldn't be required to use Monitoring · Issue #93432 · elastic/kibana · GitHub or if not, could you please file a new ticket and link here too?

Don't know what happened, I just let it be during the weekend and now even the last 4 alerts seem they have no problem now. So my issue is probably fixed, thank you!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.