Updating watchers via API doesn't take affect without manual intervention

In an attempt to get some control over our alerting we have put our watcher definitions in a GitHub repo and put together a build script so Travis deploys updates to the watchers automatically. This seems to work only only half-way in the sense that

  • the new watchers are deployed to our elasticsearch instance on elastic.co, and
  • the GUI says that the watchers are run, however
  • the watchers never actually fire

The only way we've found to "turn them on" is to manually log in to elasticsearch/kibana, and manually save each watcher.

This is not really the behaviour we expected, so we suspect there's something we are missing.

This is how we put the put a watcher on our elasticsearch instance:

put_watch() {
    local filename=$1
    local watch_name=$(basename $filename .json)

    # This is the Elasticsearch endpoint for our deployment.
    local host=<elastic.co id>.eu-central-1.aws.cloud.es.io:9243

    local url=https://$host/_xpack/watcher/watch
    http -a "<user>:<pword>" PUT $url/$watch_name < $filename

The instance in question is running version 6.2.4.

can you share the actual HTTP requests you are sending?

So far I have not seen, what is being sent to the elasticsearch node, and that would be vital for further debugging.


Of course. One of them you can find one at http://termbin.com/trqx.

thanks for the link.

this is only the request body, but does not give an indication of what was really sent. Can you run curl -v (or the equivalent of your http tool) to also see headers and responses?


Here's a full exchange, both full request and full response: http://termbin.com/b8s3.

(I should point out that I copied it from Travis, and there may be some escape codes for colours that have come through because, well, Travis kind of sucks sometimes.)

this looks fine and should execute as you expect it to be.

Did you check the watch history if no executions at all happened? Each execution of a watch has a so called watch record.

GET .watcher-history-*/_search
  "query": {
    "bool": {
      "filter": [
          "term": {
            "watch_id": "bull-prod-errors"
  "sort": [
      "trigger_event.triggered_time": {
        "order": "desc"

Does the above return anything?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.