Is it possible to have the watcher to run a shell script as an action?
It is not possible today. Watcher runs inside Elasticsearch, and we have done a lot of work to prevent new processes from being spawned from inside the server, so this isn't a feature that we're planning to add in the short term, and possibly not ever.
However, Watcher does have a webhook action that you could use to hit any service with a RESTful API, which could be a webservice that launches a shell script or runs code directly.
Just to add here, you could use the logstash exec output and the logstash http input to send the webhook to logstash.
--Alex
Thanks guys, I will take a look on those.
The reason we need this is to actually run the ACK watch API for the watcher since we do not want to be receiving any more alerts once an alarm has been sent unless it has returned to normal again during the watch. Maybe there's a way to do this internally, but I'll open a separate thread for this.