Elastic Log Threshold rule problem

amityahav · April 3, 2023, 1:19pm

Hey there, i've been experimenting with log threshold type of rules recently and i've encountered some strange behaviors. Elastic version is 8.6.2

given a log threshold rule with action connector of some index.

When disabling i would expect that a 'resolve' event would be sent to the connector for each active alert that lived while the rule was enabled. Isn't it the case?
When enabling the rule, actions are being triggered and events are written to the index, everything is working fine, but when i disable and then re-enable the alert become active again but it has no start date and duration in the kibana ui -

image1635×103 9.53 KB

and while this happens no actions are triggered, meaning no new events for my index. i let the rule run for couple of iterations and still no actions triggered. if i delete the rule and re-create it , its solves the issue.

Thanks

amityahav · April 3, 2023, 1:27pm

This is the rule:

faisal-k · April 4, 2023, 12:10pm

Hi @amityahav, welcome to the Elastic community!

Regarding your questions:

When disabling i would expect that a 'resolve' event would be sent to the connector for each active alert that lived while the rule was enabled. Isn't it the case?

Disabling the rule, will not allow the rule to rerun and check the status of the alerts, whether they are Active/Recovered. So the alerts stay at their last state and no event is sent via connectors.

When enabling the rule, actions are being triggered and events are written to the index, everything is working fine, but when i disable and then re-enable the alert become active again but it has no start date and duration in the kibana ui -

The screenshot you posted is not very clear to know from which page it has been taken, but I aussme it's from the Stack Mangement > Rules page.

For Observability rule types, I would suggest using the Alerts pages under Observability.

You would have the Alerts table with all the info you might look for.

I hope that will help!

Faisal

amityahav · April 5, 2023, 10:08am

Hey thanks for the reply,
basically what i don't understand is why after re-enabling the rule and the condition is satisfied, nothing is written to my index connector

benakansara · April 11, 2023, 10:03am

Hi @amityahav ,

There is a known issue regarding Start time and Duration being empty in the UI in the Stack Management > Rules > Rule Name > Alerts.

github.com/elastic/kibana

[ResponseOps] alert start / duration stored in the wrong slots in alert task state

opened 06:40PM - 09 Nov 22 UTC

pmuellr

bug Feature:Alerting Team:ResponseOps Feature:Alerting/RulesFramework

While looking at some live data, I happened to notice that we are storing framew…ork-required data in the alert task state, which is designed to be used only by the rule executors. Here's an example: <details> <summary>example task state JSON blob</summary> ```json { "alertInstances": { "host-1": { "state": { "start": "2022-11-08T21:34:59.929Z", "duration": "0" }, "meta": { "lastScheduledActions": { "group": "threshold met", "date": "2022-11-08T21:34:59.942Z" } } }, "host-2": { "state": { "start": "2022-11-08T21:34:59.929Z", "duration": "0" }, "meta": { "lastScheduledActions": { "group": "threshold met", "date": "2022-11-08T21:35:00.682Z" } } } }, "previousStartedAt": "2022-11-08T21:34:59.474Z" } ``` </details> The fields `alertInstances.*.state` is for rules to store alert-specific state for the next run, and `alertInstances.*.meta` is for the alerting framework to store it's own alert task state. However, the `start` and `duration` fields are generated by the framework, and should be in `meta`, not `state`. Those fields are currently populated here: https://github.com/elastic/kibana/blob/bf3ee9e393b00bce2095ccf7ad6c8693ea1837ca/x-pack/plugins/alerting/server/lib/process_alerts.ts#L70-L83 To make matters worse, rule types use the `.replaceState()` API themselves, which will usually end up removing our fields. For instance, the ES query rule does that here: https://github.com/elastic/kibana/blob/64c3c63e21009719a99e5cebe47859d0c63ce93b/x-pack/plugins/stack_alerts/server/rule_types/es_query/executor.ts#L87-L90 The result of this is that we don't write the correct start/duration to the event log, so subsequent queries against the event log won't have the data and display blank. For example, in the alerts page of a rule - the alert is listed, but there is no duration or start time. Hacking that code real quick to copy the existing `start` and `duration` into the `replaceState()` will get the start and duration to show up in that UX. Some obvious things we will need to do: - update the `meta` type to include `start` and `duration` - migration to move any existing `start` and `duration` fields from `state` to `meta` - change the code to write to `meta` instead of `state` There is likely a quicker fix if we want to get something out sooner which isn't a complete fix but would likely get the start / duration in the event log, and alerts UX working. The idea would be to modify `replaceStart()` to always copy over existing `start` and `duration` fields, if they exist in the current state. Not sure it's worth doing this, since it's not a complete fix - and I'm just guessing this would work and wouldn't cause additional problems :-)

The reason nothing is being written to index connector is that the alert is technically still active in your case even though it shows "recovered" in UI when you disable the rule. When the rule is enabled again, the same alert's status is changed to "active". It is not a new alert. Only if the status of the alert actually changed to "recovered" in the meantime (below threshold), it will be written to the index connector as per rule configuration (Notify = On status changes). You could change "Notify" option to "On check intervals" to receive notification to the index connector at every check regardless of the status (this will result in more number of notifications).

I would recommend to use Observability > Alerts to view correct start/duration of the alerts. The status of the alert remains as is (Active/Recovered) when the rule is disabled which is more accurate indication as the rule is no longer running to check the status.

I hope that helps.

-Bena