Hi there, I am quite new to the Elastic Stack, as is my company. We are attempting to build a centralized logging and alerting system, building dashboards and alerts in Kibana. Currently we have AWS Lambda functions shipping CloudWatch logs to a LogStash server, which parses our data and pushing it on to our Elasticsearch server.
Starting out simple, we have a lambda function that queries a queue and checks for events that need to be processed, and logs the result. Here's an example of one of those logs:
[INFO] 2021-01-14T17:59:22.420Z b7e95598-3c21-42bd-bbe1-cc76ddae431d $Count$: 0
Logstash then parses a couple fields, most importantly, the count into a number. I've created an OK graph of this value. I would prefer to show the exact value, but it needs an aggregation, so I instead have the sum over a period of time, which works. Anyway, not the issue here.
We want to create alerts. It is obvious and easy to setup alerts for something like, when count > 20: alert, but what we would really like to setup is more like, when count > 20 && count is growing: alert. So, for example, if we could check every half an hour, we would alert the first time the count hits 21, and then we would alert again next half hour if count had gone up to 40, or something.
The logic behind this being, that way alerts will continue until the issue is fixed. But at the same time, if the issue is fixed when count is at, say, 800, we will not get an alert next half hour when it is still over 20, but is reducing to, say to 300. Previously we had a simple threshold of 20, as described above, but we ran into an issue where, when we thought an issue was fixed, it turned out it wasn't, but we didn't get anymore alerts.
It would also be phenomenal if we could setup some sort of logic for "if this is the 7th alert, change who is emailed," but that's not essential.
If anyone can point me in the right direction of how this may be possible, it would be vastly appreciated!