While we were having some cluster issue last week I turned off Watcher using the "PUT _watcher/_stop" command. After about a week of fighting things, we got the cluster back to a place where I felt comfortable turning Watcher back on.
However, once I started it everyone started getting bombarded by emails from watcher. It appears that either it was still keeping track of all the times it SHOULD have fired while turned off, or it looked back at the period it had been turned off for and realized that no emails had been sent for that whole time and started sending them retroactively.
This particular watch is also supposed to have a 5 minute throttle, which clearly is getting ignored since since it sends off over 100 emails in less than a minute.
Any thoughts as to why this is happening and/or how to tell it not to? I'd like to get Watcher up and running again.
does the .triggered_watches index contain data/documents? If so, watches might have queued up, before you turned it off. This index contains watch runs that are supposed to be executed, but did not.
You can try to delete that index and then restart watcher.
Which version of Elasticsearch/Watcher are you using? I'd like to take a closer look about the throttling.
Thanks for the reminder about .triggered_watches! I knew there was something like that, but I just couldn't remember. We are on Elasticsearch and Watcher version 2.4 currently. It looks like the throttle was only active BEFORE they got sent into .triggered_watches. Once in that queue they just happened as fast as they could once I turned it back on.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.