How to create Alerts for cluster health (green/yellow/red) and Circuit Breaker errors?

In Kibana 8.9.0, I managed to successfully create an alert for Cluster Health,
so that if cluster health transitions from green to yellow or red,
I receive an alert.

I did the following:

  1. Go to Stack Monitoring
  2. In the top right, click on Enter setup mode
  3. In the top right, click on Alerts and rules
  4. Go to kbn:/app/management/insightsAndAlerting/triggersActions/rules
  5. Search for Cluster health

This works great.

How can I create alerts if the Circuit breaker errors
which seem to through es_rejected_execution_exception errors?

Even though my cluster health was green, I recently encountered two types of circuit breaker errors that I wish I received alerts for:

  1. Circuit Breaker error 1:
failed to publish events: 429 Too Many Requests: {"error":{"root_cause":[{"type":"es_rejected_execution_exception",
"reason":"rejected execution of coordinating operation 
[coordinating_and_primary_bytes=216741495, replica_bytes=0, all_bytes=216741495, coordinating_operation_bytes=41925,
 max_coordinating_and_primary_bytes=214748364]"}],"type":"es_rejected_execution_exception","reason":"rejected execution of coordinating operation
[coordinating_and_primary_bytes=216741495, replica_bytes=0, all_bytes=216741495, coordinating_operation_bytes=41925,
  1. Circuit Breaker error 2:
failed to index document (es_rejected_execution_exception): rejected execution of
 creationTimeNanos=775669912784792, startTimeNanos=0, finishTimeNanos=-1, failedOrRejected=false} on
 TaskExecutionTimeTrackingEsThreadPoolExecutor[name = quicknode-elastic-es-data-hot-zone-3-1/write,
queue capacity = 10000, task execution EWMA = 3.4ms, total task execution time = 58.2d,
 org.elasticsearch.common.util.concurrent.TaskExecutionTimeTrackingEsThreadPoolExecutor@2e4d252[Running, pool size = 8, active threads = 8, queued tasks = 10001, completed tasks = 639326371]]

Is there a pre-canned Alert that I can use to detect those types of errors?

If not, is there a general technique for creating alerts for Circuit Breaker errors?

As of 8.9.1 these are the rules available:

If you get those message logs ingested in your cluster properly, then you need to create custom rule based on an Elasticsearch query as documented here