Watcher cannot be started after upgrading Elasticsearch 7.x to Elasticsearch 8.x

The watcher is stop working just after the Elasticsearch has been upgraded to version 8.x.

At the Kibana's Stack Management -> Watcher, the state of all active watchers are OK, but the watchers haven't been triggered even thought the condition was met.

I have executed this API GET _watcher/stats and here is the result:

{
  "_nodes" : {
    "total" : 3,
    "successful" : 3,
    "failed" : 0
  },
  "cluster_name" : "elastic-cluster",
  "manually_stopped" : false,
  "stats" : [
    {
      "node_id" : "_Hh6Lke_RdOcox6fYy4_1w",
      "watcher_state" : "stopped",
      "watch_count" : 0,
      "execution_thread_pool" : {
        "queue_size" : 0,
        "max_size" : 0
      }
    },
    {
      "node_id" : "JwVTyt1XRcuUvpWN7YS7ig",
      "watcher_state" : "started",
      "watch_count" : 0,
      "execution_thread_pool" : {
        "queue_size" : 0,
        "max_size" : 0
      }
    },
    {
      "node_id" : "1qwaBLInRKCq2cjecv7H8w",
      "watcher_state" : "stopped",
      "watch_count" : 0,
      "execution_thread_pool" : {
        "queue_size" : 0,
        "max_size" : 0
      }
    }
  ]
}

I tried to start the watcher service by running this API POST _watcher/_start and the result was acknowledged, but there's nothing change with the status of the watcher.

First try: can you stop first, then check if all states are marked as stopped and then restart?

Second, can you take a look at Elasticsearch log files from the nodes that have not started watcher? Maybe there is something in there?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.