Thanks for the reply! The watcher work I posted this question about is on my own pre dev environment. The watcher license expired a while back, so yesterday, before posting this question, I removed watcher and the license, then downloaded/installed them again. Then I was able to create the watches, but as mentioned didn't see them being executed or any history.
Looking at the log now, I see:
ALL CONFIGURED ACTIONS ON
ALL REGISTERED WATCHES ARE THROTTLED (NOT EXECUTED) ON WATCHER LICENSE EXPIRATION.
WATCHES WILL CONTINUE BE EVALUATED AND WATCH HISTORY WILL CONTINUE BEING RECORDED.
HAVE A NEW LICENSE? PLEASE UPDATE IT. OTHERWISE, PLEASE REACH OUT TO YOUR SUPPORT CONTACT.
It doesn't look to me like the watches I created are being evaluated and there is no history, but my guess based on the language used is that the watches being referred to in the "watches will continue be evaluated" and "watch history will continue being recorded" are watches that were created before the license expired? In any case, I had already reached out to our rep about the watcher license a few days ago. Waiting to hear back.
Another question, before I checked the log, I was excited to try to execute a watch manually (didn't know about that capability before). I tried that but got "No handler found for uri [/watcher/watch/cluster_health_watch_any/execute] and method [POST]". We're currently running 1.7.2. Is that functionality available in that version? We plan to upgrade soon but being able to do this between now and then would be helpful.