Have watches created but the watches don't appear to be being triggered

Elastic Stack Elasticsearch
1

Hi,

This is the second time I'm doing this. The first time was fairly straightforward: I created the watches based on a schedule, then was able to see them occurring (saw corresponding .triggered_watches and watch_history_date indices -- I might have the underscores in the wrong places there, but you probably get my drift.). Now I'm doing this in another environment. I created the watches based on a schedule trigger, but am not seeing a triggered watches or any watch_history indices. Did I miss a step?

Including a snip of the docs in the current watcher index.

NoWatcherWatches.JPG1474×117 25.2 KB

Any help would be appreciated! Thanks!

2

Hey,

can you add some more information to this and help us to create a reproducible example?

  • Which Elasticsearch version are you using?
  • Do you have a sample watch you can share with us? What is returned when you try to store the watch?
  • What happens when you try to execute a watch manually? Can you paste the output (or put it in a gist)?
  • Are there any log entries/exceptions in the master node log of your cluster?

--Alex

1 Like
3

Thanks for the reply! The watcher work I posted this question about is on my own pre dev environment. The watcher license expired a while back, so yesterday, before posting this question, I removed watcher and the license, then downloaded/installed them again. Then I was able to create the watches, but as mentioned didn't see them being executed or any history.

Looking at the log now, I see:

 ALL CONFIGURED ACTIONS ON 
ALL REGISTERED WATCHES ARE THROTTLED (NOT EXECUTED) ON WATCHER LICENSE EXPIRATION. 
WATCHES WILL CONTINUE BE EVALUATED AND WATCH HISTORY WILL CONTINUE BEING RECORDED.
HAVE A NEW LICENSE? PLEASE UPDATE IT. OTHERWISE, PLEASE REACH OUT TO YOUR SUPPORT CONTACT.

It doesn't look to me like the watches I created are being evaluated and there is no history, but my guess based on the language used is that the watches being referred to in the "watches will continue be evaluated" and "watch history will continue being recorded" are watches that were created before the license expired? In any case, I had already reached out to our rep about the watcher license a few days ago. Waiting to hear back.

Another question, before I checked the log, I was excited to try to execute a watch manually (didn't know about that capability before). I tried that but got "No handler found for uri [/watcher/watch/cluster_health_watch_any/execute] and method [POST]". We're currently running 1.7.2. Is that functionality available in that version? We plan to upgrade soon but being able to do this between now and then would be helpful.

Thanks,
Casie

4

Hey,

putting a watch is not allowed either when your license is expired. Can you show the response of you putting a new watch?

You need to replace /watcher with /_watcher in the execute watch API to make it work.

Also, if this is a dev environment, I suggest trying to upgrade to a more recent version of Elasticsearch and Watcher if possible. Lots of features added, lots of bugs fixed. :slight_smile:

--Alex

5

Thanks for this. We got a new license. And I created new watches, or tried to, again, but I'm seeing the same thing. The reason I created the index as watcher rather than _watcher is because, when I try _watcher, I get the attached.

_watcherissue.JPG1360×58 11.7 KB

Is that the issue? That I can't create the index as _watcher and therefore the watches aren't executing?

Believe me, we want to upgrade to the latest version. There are lots of dependencies on what we've built on the current version and some other obstacles we need to overcome before we can.

6

Additionally, if it's helpful, here's what I get if I run GET /watcher/watch/cluster_health_watch_any (which is for the watch -- or an attempted watch :slight_smile: -- I created).

7

Additionally #2. Shouldn't be a license issue. When I run GET /license, I get an expiration date for watcher of Mon, 22 Aug 2016 23:59:59.999 GMT. Here's that portion of the output:

"license": {
"aliases": {},
"mappings": {},
"settings": {
"index": {
"licenses": [
{
"uid": "xxx",
"expiry_date_in_millis": "1471910399999",
"feature": "watcher",
"signature": "xxx",
"issued_to": "xxx",
"max_nodes": "15",
"subscription_type": "none",
"issue_date_in_millis": "1468972800000",
"type": "trial",
"issuer": "Aunik Bhattacharjee"
},

8

I'm going to reinstall the watcher plugin and see if that helps. Will reply back with an update as soon as I have one.

9

Hey,

this looks as if the watcher plugin was not installed properly to me. Putting sth into /watcher/watch/foo always works, because it is just creating document in the watcher index. without any of the watcher logic happening in the background though.

Can you run node stats or the hit _cat/plugins endpoint? See here

--Alex

10

Hi, again. As suspected it was an issue with the watcher plugin situation in this cluster. I removed watcher and then reinstalled the license and plugin and now the issue is resolved.

Thanks for your help! I may be back with more questions. Going to be hooking up alerts to pagerduty shortly. Going to watch the webinar on 7/28, that should be helpful.

11

Hey,

nice to hear! If you spot anything that should be documented in a better way or increased logging would have helped in your case, please mention it. Happy to improve user experience anytime...

--Alex