Hey,
around the end of october, my alerting rules stopped working.
I'm looking at the rule DNS activity to the Internet as my prime example, since this was triggering fairly regularly back when Detection Engine was still working.
It shows that the last run succeeded:
When running the query in my zeek dataset (which is in the filebeat-* index pattern, as it always has been, and which is exactly where the detection rule is looking), I get back a bunch of hits for today alone. So I know the rule should trigger.
The only recent error on that rule in the time window in question is this:
Bulk Indexing of signals failed: index: "filebeat-7.15.0-2021.10.07-000002" reason: "[REDACTED][REDACTED:9300][indices:data/read/search[phase/query]]" type: "no_shard_available_action_exception" name: "DNS Activity to the Internet" id: "0bba0dd4-3a23-4430-8542-87042badf704" rule id: "6ea71ff0-9e95-475b-9506-2580d1ce6154" signals index: ".siem-signals-default"
This occured once while I was updating the cluster to 7.15.1, and to be expected.
Through the upgrade, I've also ensured that Kibana as well as every node was restarted. This didn't fix the issue.
The .siem-signals-default index is healthy and has not been rolled over since the last successful signal was ingested.
Where could I continue troubleshooting?
PS: I think I remember that this happened before around 7.13 for me for a while. I think I remember that the Detection Engine just magically recovered after a few days.
Thanks! 