Extracting issues and slow down of watches

[seanziee (Kofi)]

I have two questions that I haven't been able to figure out from the resources available:

1. I'm trying to use extract as described in the documentation to extract one field from the input. I'm not taking out the total of hits as described in the documentation, but rather I'd like to take out a field from my document, but all my trials come to no avail.

   "input": {
   "search": {
   "request": {
   "indices": [ ".watcher-history*" ]
   },
   "extract": [ "ctx.payload.hits.hits._source.Severity" ] #I've also tried a few other iterations to no avail (hits.hits.Severity, hits.Severity, hits._source.Severity)
   }
   }

My mapping is like this when I send

GET .watcher-history*/_search?pretty

result": {
        "execution_time": "2017-02-28T15:27:11.037Z",
        "execution_duration": 3,
        "input": {
          "type": "search",
          "status": "success",
          "payload": {
            "_shards": {
              "total": 5,
              "failed": 0,
              "successful": 5
            },
            "hits": {
              "hits": [
                {
                  "_index": "logstash-2017.02.21",
                  "_type": "logs",
                  "_source": {
                    "date": "Tue, 21 Feb 2017 09:40:44 -0500",
                    "ProcessName": "example.exe",
                    "subject": "error problem",
                    "Priority": "1",
                    "Severity": "Error",
                    "PID": "09090"

How can I pull out the "Severity" from the above field in my document?

2. I want to make sure that running too many watches wont slow down my system. I'll be running approximately 30 watches running not all at the same moment but spaced out. My question is a) is that too many to run and b) On what machine do watches run on? I have a separate machine for logstash, Kibana, and ES.

Thanks a bunch!

[spinscale (Alexander Reelsen)]

Hey,

I am confused by that extract example. The watcher history does not include a Severity field. Extraction in your example does not work because you specify a concrete path with the extract parameter, but ctx.payload.hits.hits is an array and thus you would need to use something like ctx.payload.hits.hits.0._source.Severity

on your second question. Currently watches are executed on the master node of your elasticsearch cluster. Given the low number of watches and those being spaced out, I would not be worried about performance for now.

--Alex

[seanziee (Kofi)]

For the extract example, my logs have a field "Severity" that I'd like to extract only (I have about 100 other fields) that get in the way that I don't actually use in the watch. I guess my broader question is how does the extract field work because it seems to only really work with "hits.total" and not with any of the data actually present in the document. Your suggestion of input still causes an error:

      "input": {
    "type": "search",
    "status": "failure",
    "reason": "NullPointerException[null]"
  },
  "actions": []
},
"messages": [
  "failed to execute watch input"

I don't want to pull out of the first document's Severity, I want that for every document that matches the query, I only get the Severity field and ignore the rest (for efficiency) . How do I form the extract field to do that?

And good to know that 30 watches is considered a little

[spinscale (Alexander Reelsen)]

Hey,

if you are using the search input, you could go with source filtering instead. See this example

PUT foo/bar/_bulk
{ "index" : {} }
{ "foo" : "bar", "severity" : "low" }
{ "index" : {} }
{ "foo" : "bar", "severity" : "high" }
{ "index" : {} }
{ "foo" : "bar", "severity" : "medium" }

PUT _xpack/watcher/watch/foo
{
  "trigger" : {
    "schedule" : {
      "interval": "10m"
    }
  },
  "input" : {
    "search": {
      "request" : {
        "indices" : ["foo"],
        "body" : {
          "query" : {
            "match_all": {}
          },
          "_source" : "severity" 
        }
      }
    }
  },
  "actions" : {
    "logging" : {
      "logging" : {
        "text" : "{{ctx.payload}}"
      }
    }
  }
}

POST _xpack/watcher/watch/foo/_execute

when executing the watch, you wont see the foo field in the search responses

--Alex