Hi @ShadwDrgn
Little followup, for when you get back to this I did test and it did appear to work OK /As expected. That is not to say there could still be an issue somewhere.
This framework is all fairly new over the last several release, and there were certainly bugs and there are still a few quirks and the docs are evolving. This caught my eye because I helped identify a few bugs in those early releases and in 7.11.0.
That said looks like it is working, ping back if you have issues.
I / we appreciate your feedback when try it especially for the first time, if there are things we can make better please let us know.
Below you can see my config.
I think there are a couple things that may give the appearance of not working right.
First, When the alert first gets saved / submitting it take a little bit of time for it to start executing.
2nd if metricbeat / elastic agent is set to collect filesystem every 1min and the Alert Checks Every 1min and it Needs to be over the Threshold for 1min its possible it can be just less than 2 mins before the first alert actually fires after it gets put in the queue to execute. The same will hold true for the recovered alert.
Also the little graph needs a little tuning I think, sometimes it is hard to interpret, it has a 20 min and yet we are trying to see 1 min... if that make sense, good for a trend, perhaps not for a specific min etc.
In the real world when you create an Metric Alert even if you filter you will probably want to do Alert Per host.name and system.filesystem.device_name (or whatever) so those terms show up in the Alert Context, if you only do a filter... the filter does not show up in the alert context.
I have included what my alert looks like plus my alert Fired and Recovered Message looks like some reason we don't have a 1 click way to do that.
These are my actions
{
"message" : "FIRED Filesystem Alert",
"alertActionGroup" : "{{alertActionGroup}}",
"alertActionGroupName" : "{{alertActionGroupName}}",
"alertId" : "{{alertId}}",
"alertInstanceId" : "{{alertInstanceId}}",
"alertName" : "{{alertName}}",
"context.alertState" : "{{context.alertState}}",
"context.group" : "{{context.group}}",
"context.metric" : "{{context.metric}}",
"context.reason" : "{{context.reason}}",
"context.threshold" : "{{context.threshold}}",
"context.timestamp" : "{{context.timestamp}}",
"context.value" : "{{context.value}}",
"date" : "{{date}}",
"spaceId" : "{{spaceId}}",
"tags" : "{{tags}}"
}
{
"message" : "RESOLVED Filesystem Alert",
"alertActionGroup" : "{{alertActionGroup}}",
"alertActionGroupName" : "{{alertActionGroupName}}",
"alertId" : "{{alertId}}",
"alertInstanceId" : "{{alertInstanceId}}",
"alertName" : "{{alertName}}",
"context.alertState" : "{{context.alertState}}",
"context.group" : "{{context.group}}",
"context.metric" : "{{context.metric}}",
"context.reason" : "{{context.reason}}",
"context.threshold" : "{{context.threshold}}",
"context.timestamp" : "{{context.timestamp}}",
"context.value" : "{{context.value}}",
"date" : "{{date}}",
"spaceId" : "{{spaceId}}",
"tags" : "{{tags}}"
}
And this is what a result looks like....
it does bother me that the host.name does not show up in the context.group and a few other fields are missing in the resolved, I think I will write that up, but it does show in the alertInstanceId and the 2 but the 2 alerts are tied by the alertId as well.
{
"message" : "FIRED Filesystem Alert",
"alertActionGroup" : "metrics.threshold.fired",
"alertActionGroupName" : "Fired",
"alertId" : "944b7010-8528-11eb-a8a7-9feb424b7003",
"alertInstanceId" : "stephenb-metrics-test, /dev/sda1",
"alertName" : "test-filesytem",
"context.alertState" : "ALERT",
"context.group" : "stephenb-metrics-test, /dev/sda1",
"context.metric" : "{"condition0":"system.filesystem.used.pct"}",
"context.reason" : "system.filesystem.used.pct is greater than a threshold of 25% (current value is 28.4%)",
"context.threshold" : "{"condition0":["25%"]}",
"context.timestamp" : "2021-03-15T01:24:00.000Z",
"context.value" : "{"condition0":"28.4%"}",
"date" : "2021-03-15T01:25:05.205Z",
"spaceId" : "default",
"tags" : ""
}
{
"message" : "RESOLVED Filesystem Alert",
"alertActionGroup" : "recovered",
"alertActionGroupName" : "Recovered",
"alertId" : "944b7010-8528-11eb-a8a7-9feb424b7003",
"alertInstanceId" : "stephenb-metrics-test, /dev/sda1",
"alertName" : "test-filesytem",
"context.alertState" : "",
"context.group" : "",
"context.metric" : "",
"context.reason" : "",
"context.threshold" : "",
"context.timestamp" : "",
"context.value" : "",
"date" : "2021-03-15T01:28:14.413Z",
"spaceId" : "default",
"tags" : ""
}