Hey @stephenb Thanks for all of this. I haven't had time to get back to this as yet, but I will say that one slightly frustrating thing was that a lot of data is missing from the recovery. suspect I understand why this is, but it would be nice if some of the context data were present in the recovery data. for example:
context.alertState is empty when it should probably be "OKAY" or "RECOVERED" or something like that
context.threshold should still contain the threshold for the alert so that I can say in my recovery "the metric has recovered from passing the {{context.threshold}}" or something.
context.metric would be good to have in a recovery too so that people know what is being restored. for example one might have a cpu AND a memory alert that's recently triggered for the same machine, but they would have to basically look at the GUID to know which thing recovered.
Finally the only way I've seen so far (on brief inspection admittedly) of discovering what is recovering in general for an alert is alertInstanceId which I have to put in both the alert and recovery so that people know what recovered.
If context.group is supposed to contain which host (if alerting separately by host.name for example) alerted, then it should also be present when that particular host recovers.
All of this is of course predicated on the assumption that this is all still true in .2 as i am still on .1
I should be able to write one payload that is identical for both alerts and recoveries eg:
{{context.alertState}}}: {{alertName}} on {{context.group}} threshold: {{context.threshold}} current value {context.value}}\n\n{{msg}}
This would allow you to also cleanly write to other indexes in a manner that would allow very easy alerting reports/metadata/dashboards or craft emails that are consistent. I'm sure there would be other advantages I'm not thinking of to added data. 