SIEM Rule Failures

Hello Everyone,

We are on Elastic 7.9 and are mainly using it as a SIEM, suddenly all of the SIEM rules start to fail and not just some but all of them.

Hi @Ameer_Mukadam! Looking at your screenshot, it appears that these failures are due to a large amount of time passing between rule executions.

Before we dig into potential causes/solutions, I want to note two important facts:

  1. Rule failures do not necessarily mean that the rule did not execute and generate signals; in fact in your case I would expect that signals are still being generated.
  2. We only report this particular error if the gap is 4x the rule interval, so given your failure messages I am inferring that your rules run every 1-2 minutes.

That being said, the failures you're seeing are indicative of a performance issue: the amount of time it takes for these rules to query and generate signals is > 4x the expected interval.

While we offer documentation for tuning our prebuilt rules, and many knobs to tune your custom rules (scheduling, query optimization, # of task workers, general vertical/horizontal scaling, etc) , these are mostly manual and depend strongly on your particular environment.

If you have a specific question or more details about the circumstances of your situation I'd be happy to help further!

I think I agree with the performance part the cluster isn’t quick at all, so that might be the issue. Also the rules run at 5 min intervals with 1-2 minutes look back time.