Elasticsearch & Kibana version: 6.2.4
Hi,
This morning we noticed that several (but not all) of our watches were not triggering and had no watcher history. In particular, one watch should have been firing alerts off but was totally silent. Using POST _xpack/watch/{watch_id}/_execute to manually trigger the alert caused it to fire.
Stopping and starting watcher with POST _xpack/watcher/_stop & POST _xpack/watcher/_start caused all watches to resume triggering. Watcher histories are being recorded again as well.
This seems similar to the issue encountered here, and I am wondering if this is a known bug in X-Pack? Watches are working as expected now, but without figuring out what caused this I can't be sure it won't happen again.
Here are some details on the cluster:
Five nodes running Elasticsearch 6.2.4, one 6.2.4 Kibana instance, one 6.2.4 Logstash instance. We upgraded from 3 nodes to a 5 node cluster a few weeks ago, I am not sure if this coincides with when the watches stopped.
Result of GET_xpack/watcher/stats/ (done after restarting Watcher, not sure how much help it will be):
{
"_nodes": {
"total": 5,
"successful": 5,
"failed": 0
},
"cluster_name": "clustername",
"manually_stopped": false,
"stats": [
{
"node_id": "L8C9e9FgSTulsHHs7vEqLQ",
"watcher_state": "started",
"watch_count": 12,
"execution_thread_pool": {
"queue_size": 0,
"max_size": 20
}
},
{
"node_id": "fdKsVvaFQ9i6er6ck5eoVQ",
"watcher_state": "started",
"watch_count": 15,
"execution_thread_pool": {
"queue_size": 0,
"max_size": 20
}
},
{
"node_id": "RskSddGqQ26Doz6y4mmTsg",
"watcher_state": "started",
"watch_count": 0,
"execution_thread_pool": {
"queue_size": 0,
"max_size": 0
}
},
{
"node_id": "C2tmZvL8QbWZ1CRZCHGWPQ",
"watcher_state": "started",
"watch_count": 0,
"execution_thread_pool": {
"queue_size": 0,
"max_size": 20
}
},
{
"node_id": "J043eFCoSV65Pd5vT9XERQ",
"watcher_state": "started",
"watch_count": 0,
"execution_thread_pool": {
"queue_size": 0,
"max_size": 0
}
}
]
}
elasticsearch.yml:
cluster.name: clustername
node.name: ch-elas-epm01-p
path.data: /data/elasticsearch/data
network.host: 0.0.0.0
bootstrap.memory_lock: true
http.port: 9200
xpack.security.enabled: false
thread_pool.search.queue_size: 10000
discovery.zen.ping.unicast.hosts: ["ch-elas-epm01-p","ch-elas-epm02-p","ch-elas-epm03-p","ch-elas-epm06-p","ch-elas-epm07-p"]
discovery.zen.minimum_master_nodes: 3
path.logs: /data/elasticsearch/logs
action.destructive_requires_name: true
xpack.notification.email:
account:
smtp_acct:
profile: standard
smtp.host: smtpserverhost
smtp.auth: false
smtp.user: no-reply@companydomain.com
default_account: smtp_acct