Custom Threshold not triggering alert to index

I tried to create a rule using custom threshold to write to an index for the alert action.
Running 8.13.

  • I created the index, and mappings ahead of time
  • I added the connector + the index
  • I tested the rule by going below the threshold, I see the alert triggers in the rule (But the index never gets populated) Contents of action trigger here:
    {
      "@timestamp": "{{date}}",
      "alert_id": "{{alert.id}}",
      "rule_name": "{{rule.name}}",
      "foo": "bar"
    }
    
  • I tested the connector by running a test, and the index gets populated each time I do.
  • I tried creating new indexes and rules, same problem every time.
  • I made sure I had correct roles + spaces enabled (maybe I missed something here?)
  • I tried creating a rule that used "log threshold" instead. This actually works, and the index gets a document per alert.
  • I tried all of this on a separate cluster, same results.

No matter what, the alert refuses to trigger the action.
Why do custom thresholds not trigger index actions?

I upgraded to 8.18. same problem.
The only thing that seems to work is setting the alert action frequency from "For each alert - on status changes" to "Summary of alerts - on check intervals"
But this will just keep sending an alert to the index if the alert is triggered.
I don't want that.
I want just one alert sent to the index if the alert happens. And once its resolved send another alert.
Does anyone else have this issue? Am I the only one?
Can anyone help please?

I created a ticket for this on Kibana github, a while back. I just updated the post here