Watcher query returns nothing, works on manual search?

[ryanlutgen]

Hello,

I am attempting to create a watcher with ES/Xpack 5.6.2 that will send a notification if an Xpack Security user count threshold is reached. The watcher looks like this, note I have the interval set to 10s for testing:

{
  "trigger": {
    "schedule": {
      "interval": "10s"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          ".security"
        ],
        "types": [
          "user"
        ],
        "body": {
          "query": {
            "match_all": {}
          }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "gt": "{{ctx.metadata.maxAccounts}}"
      }
    }
  },
  "actions": {
    "send_email": {
      "email": {
        "profile": "outlook",
        "to": [
          "..."
        ],
        "subject": "...",
        "body": {
          "text": "..."
        }
      }
    }
  },
  "metadata": {
    "cluster_name": "...",
    "maxAccounts": 5
  }
}

I have around 30 accounts on my test cluster, so I should be causing this watcher to fire every 10 seconds, however nothing happens. I have other watchers on the cluster that do successfully send emails, so I don't think its an issue with email configuration in the elasticsearch yml. Attempting to simulate the watch, I see that there are no hits returned in the payload.

I execute the search query manually and I get the expected results:

GET .security/user/_search
{
  "query": {
    "match_all": {}
  }
}

I attempted to change the watch to just an an existence check for usernames, omitting the type in the watcher config, but it also yields no results when simulating:

...
"result": {
    "execution_time": "2017-10-18T18:57:42.470Z",
    "execution_duration": 0,
    "input": {
      "type": "search",
      "status": "success",
      "payload": {
        "_shards": {
          "total": 0,
          "failed": 0,
          "successful": 0,
          "skipped": 0
        },
        "hits": {
          "hits": [],
          "total": 0,
          "max_score": 0
        },
        "took": 0,
        "timed_out": false
      },
      "search": {
        "request": {
          "search_type": "query_then_fetch",
          "indices": [
            ".security"
          ],
          "types": [],
          "body": {
            "query": {
              "exists": {
                "field": "username"
              }
            }
          }
        }
      }
    },
    "condition": {
      "type": "compare",
      "status": "success",
      "met": false,
      "compare": {
        "resolved_values": {
          "ctx.metadata.maxAccounts": 5,
          "ctx.payload.hits.total": 0
        }
      }
    },
    "actions": []
  }
...

There are no errors in the logs during the time the watcher is executing.

I had this working at some point on ES/Xpack 5.5.1, but I don't recall anything changing, other than the version we are running our test cluster on. I forget the exact timing of when we upgraded relative to when I first noticed this, I think it might have stopped working before we upgraded, but again I am not 100% sure.

Anybody have a clue on why this isn't working like I'd expect?

[spinscale (Alexander Reelsen)]

Hey

That index is not accessible when you execute a watch due to security reasons.