Can't create a new Watch

Elastic Stack Elasticsearch
1

Hi - new to Watcher. Just installed it using the "Existing Cluster" instructions.

We recently upgraded to ElasticSearch 2.3.1.

When I run XGET /_watcher/stats?pretty, I get the following, which seems encouraging:
{
"watcher_state": "started",
"watch_count": 0,
"execution_thread_pool": {
"queue_size": 0,
"max_size": 0
},
"manually_stopped": false
}

I've also confirmed that the .watches index was created, per GET /_cat/indices?v. However, the index status is red.

When I try to create a new Watch, I get the following error:
{
"error": {
"root_cause": [
{
"type": "timeout_exception",
"reason": "Timeout waiting for task."
}
],
"type": "timeout_exception",
"reason": "Timeout waiting for task."
},
"status": 500
}

I've restarted ElasticSearch after each step. I've also tried uninstalling and reinstalling Watcher, with the same results.

Any idea where I went wrong?

2

Hey,

if the index status is red, it was not correctly recovered on startup and thus no watches can be executed. Do you find any information about that in the master node logs? Any stack traces or exceptions giving a hint what happened to this index?

--Alex