Updating watchers via API doesn't take affect without manual intervention

[Magnus_Therning]

In an attempt to get some control over our alerting we have put our watcher definitions in a GitHub repo and put together a build script so Travis deploys updates to the watchers automatically. This seems to work only only half-way in the sense that

• the new watchers are deployed to our elasticsearch instance on elastic.co, and
• the GUI says that the watchers are run, however
• the watchers never actually fire

The only way we've found to "turn them on" is to manually log in to elasticsearch/kibana, and manually save each watcher.

This is not really the behaviour we expected, so we suspect there's something we are missing.

This is how we put the put a watcher on our elasticsearch instance:

put_watch() {
    local filename=$1
    local watch_name=$(basename $filename .json)

    # This is the Elasticsearch endpoint for our deployment.
    local host=<elastic.co id>.eu-central-1.aws.cloud.es.io:9243

    local url=https://$host/_xpack/watcher/watch
    http -a "<user>:<pword>" PUT $url/$watch_name < $filename
}

The instance in question is running version 6.2.4.

[spinscale (Alexander Reelsen)]

can you share the actual HTTP requests you are sending?

So far I have not seen, what is being sent to the elasticsearch node, and that would be vital for further debugging.

--Alex

[Magnus_Therning]

Of course. One of them you can find one at http://termbin.com/trqx.

[spinscale (Alexander Reelsen)]

thanks for the link.

this is only the request body, but does not give an indication of what was really sent. Can you run curl -v (or the equivalent of your http tool) to also see headers and responses?

Thanks!

[Magnus_Therning]

Here's a full exchange, both full request and full response: http://termbin.com/b8s3.

(I should point out that I copied it from Travis, and there may be some escape codes for colours that have come through because, well, Travis kind of sucks sometimes.)

[spinscale (Alexander Reelsen)]

this looks fine and should execute as you expect it to be.

Did you check the watch history if no executions at all happened? Each execution of a watch has a so called watch record.

GET .watcher-history-*/_search
{
  "query": {
    "bool": {
      "filter": [
        {
          "term": {
            "watch_id": "bull-prod-errors"
          }
        }
      ]
    }
  },
  "sort": [
    {
      "trigger_event.triggered_time": {
        "order": "desc"
      }
    }
  ]
}

Does the above return anything?