Elasticsearch v. 6.2.4
Context: I have aggregated some data into buckets. I want the derivative of the data and also stabilize the derivative with a moving average.
Problem: When I chain the derivative to a moving average aggregation, the results are delayed by one bucket. It still works if I chain two derivatives in order to get a 2nd order derivative. Are there any reasons to why it works when chaining two derivatives and not a moving average?
Edit: Some clarification, the moving average does not include the derivative at its current bucket.
Note:
- burndown is balance_change without positive values
- second_order still works properly with either burndown or balance_change as path. Moving average does not.
Here is the request body
{
"size": 0,
"query": {
"match": {
"user_key": user_key
}
},
"aggs": {
"history": {
"nested": {
"path": "history"
},
"aggs": {
"time_range": {
"filter": {
"range": {
"history.request_time": {
"gte": "2018-04-13T06:30:39",
"lte": "2018-05-20T06:30:39"
}
}
},
"aggs": {
"time_bucket": {
"date_histogram": {
"field": "history.request_time",
"interval": "day"
},
"aggs": {
"bytes": {
"max": {
"field": "history.remainingBytes"
}
},
"balance_change": {
"derivative": {
"buckets_path": "bytes",
"unit": "day"
}
},
"burndown": {
"bucket_script": {
"buckets_path": {
"balance_change": "balance_change.normalized_value"
},
"script": """
if (params.balance_change > 0) {return null;}
else {
return params.balance_change;
}
"""
}
},
"moving_average_of_burndown": {
"moving_avg": {
"buckets_path": "burndown",
"gap_policy": "insert_zeros",
"window": 7,
"model": "ewma",
"settings": {
"alpha": 0.5
}
}
},
"second_order": {
"derivative": {
"buckets_path": "balance_change"
}
}
}
}
}
}
}
}
}
}
Here is a plot of my results
