hi guys I have a weird issue.
I have number of alerts which are active only for very brief time window say for 30 minutes to 2 hour other time they are snoozed via snooze shedule. However since last 2-3 days I am getting notifications at snoozed time window which must not be the case. We have not changed these alert rules and associated query for almost 6+ months. So it is very weird. We are not able to understand this behavior.
We also noticed on alerts rules listing page, red highlight is missing on snoozed alerts but when you open the schedule - it does have the right snooze schedule.
Something to note here is : when the alert were setup, we had Elastic and Kibana v8.6.x but now almost a month back, we upgraded to 8.8.2 without any problems. This weird behaviour around snooze only seen in last 3 days.
We also have license renewal due in 25 days.
Any guidance in triaging this issue will be greatly appreciated. Tagging @ Shani_Sagiv as if you are aware of anything like this!