While we were having some cluster issue last week I turned off Watcher using the "PUT _watcher/_stop" command. After about a week of fighting things, we got the cluster back to a place where I felt comfortable turning Watcher back on.
However, once I started it everyone started getting bombarded by emails from watcher. It appears that either it was still keeping track of all the times it SHOULD have fired while turned off, or it looked back at the period it had been turned off for and realized that no emails had been sent for that whole time and started sending them retroactively.
This particular watch is also supposed to have a 5 minute throttle, which clearly is getting ignored since since it sends off over 100 emails in less than a minute.
Any thoughts as to why this is happening and/or how to tell it not to? I'd like to get Watcher up and running again.