Alerting on cluster (watcher)

[sahere37 (LR)]

hi all,
I am using filebeat, elasticsearch, kibana and x-pack. Version of all is 6.1.1.
I defined watch, condition and action as following:

PUT _xpack/watcher/watch/log_error_watch
{
 "trigger" : { "schedule" : { "interval" : "10s" }},
 "input" : {
 "search" : {
  "request" : {
    "indices" : [ "myind" ],
    "body" : {
      "query" : {
        "match" : { "message": "memory" }
      }
    }
  }
}
},
"condition" : {
"compare" : { "ctx.payload.hits.total" : { "gt" : 0 }}
},
"actions" : {
 "log_error" : {
  "logging" : {
    "text" : "Found {{ctx.payload.hits.total}} memory phrases in the myind 
 index"
       }
 }
}
}

But, there is no message in the elasticsearch log. To check the state of watcher, i used following command:

 curl -XGET 'http://192.168.170.152:9200/.watch_history*/_search?pretty'

Which the result is as following. Notably, i installed x-pack, other features like security, role and users are OK and there is no "missing authentication" error.

 # curl -XGET 'http://192.168.170.152:9200/.watch_history*/_search? 
 pretty'
{
    "error" : {
    "root_cause" : [
      {
       "type" : "security_exception",
       "reason" : "missing authentication token for REST request 
    [/.watch_history*/_search?pretty]",
       "header" : {
         "WWW-Authenticate" : "Basic realm=\"security\" charset=\"UTF-8\""
       }
     }
   ],
     "type" : "security_exception",
     "reason" : "missing authentication token for REST request 
 [/.watch_history*/_search?pretty]",
    "header" : {
    "WWW-Authenticate" : "Basic realm=\"security\" charset=\"UTF-8\""
  }
},
"status" : 401
}

Any solution will be so appreciated. thanks.

[baz (Michael Basnight)]

Hi, I see two things happening here. One, that watcher is not firing, or you cannot tell if it is. two, you cannot access your cluster. When you turn on a trial license and enable security, the next steps you need to do are to set up a few users, here is the tutorial for that.

Once you have gotten thru that and see data being returned to you, the next thing would be to run the PUT again on the watch with the authenticated user. You can also test your watches with _execute watch API, so you dont have to sit there and let it run, the output of what would happen will be displayed to you, as well as any indicating information about whether it met the condition you have put in the code as well. It is the most powerful feature of watcher since it allows you to tweak your watches until they are perfect. Docs for it are here