Elastic Log Threshold rule problem

Elastic Observability
1

Hey there, i've been experimenting with log threshold type of rules recently and i've encountered some strange behaviors. Elastic version is 8.6.2

given a log threshold rule with action connector of some index.

  1. When disabling i would expect that a 'resolve' event would be sent to the connector for each active alert that lived while the rule was enabled. Isn't it the case?

  2. When enabling the rule, actions are being triggered and events are written to the index, everything is working fine, but when i disable and then re-enable the alert become active again but it has no start date and duration in the kibana ui -

    image
    image1635×103 9.53 KB

    and while this happens no actions are triggered, meaning no new events for my index. i let the rule run for couple of iterations and still no actions triggered. if i delete the rule and re-create it , its solves the issue.

Thanks

2

This is the rule:

image
image598×634 49.2 KB

image
image588×552 64.7 KB

image
image596×520 83.7 KB

3

Hi @amityahav, welcome to the Elastic community!

Regarding your questions:

  1. When disabling i would expect that a 'resolve' event would be sent to the connector for each active alert that lived while the rule was enabled. Isn't it the case?

Disabling the rule, will not allow the rule to rerun and check the status of the alerts, whether they are Active/Recovered. So the alerts stay at their last state and no event is sent via connectors.

  1. When enabling the rule, actions are being triggered and events are written to the index, everything is working fine, but when i disable and then re-enable the alert become active again but it has no start date and duration in the kibana ui -

The screenshot you posted is not very clear to know from which page it has been taken, but I aussme it's from the Stack Mangement > Rules page.

For Observability rule types, I would suggest using the Alerts pages under Observability.

Screenshot 2023-04-04 at 14.03.13
Screenshot 2023-04-04 at 14.03.13452×636 19.7 KB

You would have the Alerts table with all the info you might look for.

Screenshot 2023-04-04 at 14.03.59
Screenshot 2023-04-04 at 14.03.591576×932 129 KB

I hope that will help!

Faisal

4

Hey thanks for the reply,
basically what i don't understand is why after re-enabling the rule and the condition is satisfied, nothing is written to my index connector

5

Hi @amityahav ,

There is a known issue regarding Start time and Duration being empty in the UI in the Stack Management > Rules > Rule Name > Alerts.

The reason nothing is being written to index connector is that the alert is technically still active in your case even though it shows "recovered" in UI when you disable the rule. When the rule is enabled again, the same alert's status is changed to "active". It is not a new alert. Only if the status of the alert actually changed to "recovered" in the meantime (below threshold), it will be written to the index connector as per rule configuration (Notify = On status changes). You could change "Notify" option to "On check intervals" to receive notification to the index connector at every check regardless of the status (this will result in more number of notifications).

I would recommend to use Observability > Alerts to view correct start/duration of the alerts. The status of the alert remains as is (Active/Recovered) when the rule is disabled which is more accurate indication as the rule is no longer running to check the status.

I hope that helps.

-Bena

1 Like