Watcher alert on buckets data

[Anabella_Cristaldi (Anna)]

Hello all,
Sorry if my question is "too basic", but I'm stuck.
I'm implementing a prototype in order to detect fraudulent phone calls.
I have made a query that given a time interval, returns the total duration of all the phone calls that have started and ended in such interval, by originator
The query (probably it can be improved) returns the desired results and now I need to alert over the query results.

If the "grand_total" is bigger than the duration of interval, then I need to raise an alert.
Based on the approach explained in

Implementing a Statistical Anomaly Detector in Elasticsearch - Part 3

I have tried to extract the relevant data for my alert that is the "key" and the "grand_total" without success.
In the simulation I get the following error

"Watcher: [parse_exception] could not parse input for watch [inlined]. expected an object representing input [extract], but found [START_ARRAY] instead"
Here is the code

{
"trigger": {
"schedule": {
"interval": "30m"
}
},
"input": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"new_sbc"
],
"types": [],
"body": {
"query": {
"bool": {
"filter": [
{
"terms": {
"oper.keyword": [
"START",
"STOP"
]
}
},
{
"range": {
"@timestamp": {
"gte": "2017-10-04T20:46:00.000Z",
"lte": "2017-10-04T20:49:10.000Z"
}
}
}
]
}
},
"aggs": {
"origin": {
"terms": {
"field": "from.keyword",
"size": 10000,
"min_doc_count": 2
},
"aggs": {
"call_ref": {
"terms": {
"field": "call_ref.keyword",
"size": 10000,
"min_doc_count": 2
},
"aggs": {
"total_duration": {
"sum": {
"field": "duration"
}
}
}
},
"grand_total": {
"sum_bucket": {
"buckets_path": "call_ref>total_duration"
}
}
}
}
}
}
}
},
"extract": [
"aggregations.origin.buckets.key",
"aggregations.origin.buckets.grand_total"
]
},
"condition": {
"always": {}
},
"actions": {
"my-logging-action": {
"logging": {
"level": "info",
"text": "There are {{ctx.payload.aggregations}} documents in your index."
}
}
}
}

I know that the information is there but I don't realize how can I access to each bucket field, either in the condition section nor in a action section.
If I remove the extract statement, doing a simulation I can see the buckets in the logging section.

{origin={doc_count_error_upper_bound=0, sum_other_doc_count=0, buckets=[{doc_count=7, call_ref={doc_count_error_upper_bound=0, sum_other_doc_count=0, buckets=[]}, grand_total={value=0.0}, key=3761582160}, {doc_count=7, call_ref={doc_count_error_upper_bound=0, sum_other_doc_count=0, buckets=[{doc_count=2, total_duration={value=5.0}, key=p65539t1507142822m18319c57997507s2}, {doc_count=2, total_duration={value=6.0}, key=p65539t1507142871m863070c57997494s2}, {doc_count=2, total_duration={value=235.0}, key=p65539t1507142906m435705c57997798s2}]}, grand_total={value=246.0}, key=376822289}, ...............

Any help will be appreciated
Regards
Anna

[lueneburger (Dirk L.)]

HI @Anabella_Cristaldi ,

I'm a little bit confused about this

so you don't actually want to build a statistical anomaly detector, you just used it as an example for your watch?
I would try to remove the "extract" part and change the condition to something like:

..aggregations

  },
  "condition": {
    "compare": {
      "ctx.payload.aggregations.grand_total.value": {
        "lt": "{{ctx.payload.aggregations.buckets.key}}"
      }
    }
  },

....action

something like that and take a look here:
compare condition

Cheers,
Dirk

[Anabella_Cristaldi (Anna)]

Hi @lueneburger,

Yes, you are correct. I used the anomaly detector as an example for my watch.
I removed the extract and tried the compare condition but still not working (ctx.payload.aggregations.grand_total.value and ctx.payload.aggregations.buckets.key are evaluated to null).

What I need is to know how to loop over the buckets in the result, access to the bucket fields (grand_total value for example) and compare against some threshold,

Another thing that is interesting in the anomaly detector example is that they first create a new index with the "interesting" values and then alert over document on that index; but first I need how to loop and access to the values in the result.

Thank you very much!
Regards
Ana

[lueneburger (Dirk L.)]

Hi @Anabella_Cristaldi,

could you provide a Simulation Result without the extract, you copy & paste it here and mark all the output here and press Ctrl + Shift + C, then you will keep the format and its easier to read

Cheers,
Dirk

[Anabella_Cristaldi (Anna)]

Thanks @lueneburger for your suggestion of using Array Compare Condition. It worked like a charm.

"array_compare": {
  "ctx.payload.aggregations.origin.buckets": {
    "path": "grand_total.value",
    "gt": {
      "value": 340,
      "quantifier": "some"
    }
  }
}

},

I have also tried an script condition that worked ok.

"condition": {
"script": {
"source": "def status= false;def calls=ctx.payload.aggregations.origin;for (int i = 0; i < calls.size(); ++i) {if (calls.buckets[i].grand_total.value > 346){ status=true;return status}} return false",
"lang": "painless"
}

Thanks!
Regards
Anna