Problem extracting buckets

Elastic Stack Elasticsearch
1

I've been using the excellent engineering blog posts on the atlas algorithm using watcher:

And I'm trying to implement something similar here. I am basically trying to track operation calls and their duration over a sliding 24 hour window.

The query works and returns data in this format:

{
"took": 89,
"timed_out": false,
"_shards": {
"total": 495,
"successful": 495,
"failed": 0
},
"hits": {
"total": 14776,
"max_score": 0,
"hits": []
},
"aggregations": {
"metrics": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 123,
"buckets": [
{
"key": "TEST_OPERATION",
"doc_count": 4884,
"queries": {...}
"ninetieth_surprise": {
"values": {
"90.0": 1110.8029139975035
}
}
}

However when I run the following watch, the buckets that are returned are empty. Any ideas? Thanks!

{
"trigger":{
"schedule":{
"interval":"1m"
}
},
"input":{
"search":{
"request":{
"indices":[
"test*"
],
"body":{
"query":{...},
"size":0,
"aggs":{
"metrics":{
"terms":{
"field":"operationname.raw"
},
"aggs":{
"queries":{
"terms":{
"field":"operationname.raw"
},
"aggs":{
"series":{
"date_histogram":{
"field":"lastmodified",
"interval":"hour",
"min_doc_count":0
},
"aggs":{
"avg":{
"avg":{
"field":"duration"
}
},
"movavg":{
"moving_avg":{
"buckets_path":"avg",
"window":24,
"model":"simple"
}
},
"surprise":{
"bucket_script":{
"buckets_path":{
"avg":"avg",
"movavg":"movavg"
},
"script":"(avg - movavg).abs()"
}
}
}
},
"largest_surprise":{
"max_bucket":{
"buckets_path":"series.surprise"
}
}
}
},
"ninetieth_surprise":{
"percentiles_bucket":{
"buckets_path":"queries>largest_surprise",
"percents":[
90.0
]
}
}
}
}
}
}
},
"extract":[
"aggregations.metrics.buckets.key",
"aggregations.metrics.buckets.ninetieth_surprise"
]
}
},
"actions":{...}
}

Thanks again.

2

Hey,

two things here.

First: You can use the execute Watch API or the watch history to check what has been returned from your search requests - this will show you if the search response looks like excpected.

Second: Using the extract feature, you might actually want to combine this with the keyed feature of the percentile aggregation, so that you can specify the full path, see the range aggs docs (this will also work for the percentile agg).

Hope this helps. Otherwise feel free to share the output of the execute API.

--Alex