Chaining derivative aggregation to moving average is delayed by one bucket


(Uy) #1

Elasticsearch v. 6.2.4

Context: I have aggregated some data into buckets. I want the derivative of the data and also stabilize the derivative with a moving average.

Problem: When I chain the derivative to a moving average aggregation, the results are delayed by one bucket. It still works if I chain two derivatives in order to get a 2nd order derivative. Are there any reasons to why it works when chaining two derivatives and not a moving average?
Edit: Some clarification, the moving average does not include the derivative at its current bucket.

Note:

  • burndown is balance_change without positive values
  • second_order still works properly with either burndown or balance_change as path. Moving average does not.

Here is the request body

{
        "size": 0,
        "query": {
            "match": {
                "user_key": user_key
            }
        },
        "aggs": {
            "history": {
                "nested": {
                    "path": "history"
                },
                "aggs": {
                    "time_range": {
                        "filter": {
                            "range": {
                                "history.request_time": {
                                    "gte": "2018-04-13T06:30:39",
                                    "lte": "2018-05-20T06:30:39"
                                }
                            }
                        },
                        "aggs": {
                            "time_bucket": {
                                "date_histogram": {
                                    "field": "history.request_time",
                                    "interval": "day"
                                },
                                "aggs": {
                                    "bytes": {
                                        "max": {
                                            "field": "history.remainingBytes"
                                        }
                                    },
                                    "balance_change": {
                                        "derivative": {
                                            "buckets_path": "bytes",
                                            "unit": "day"
                                        }
                                    },
                                    "burndown": {
                                        "bucket_script": {
                                            "buckets_path": {
                                                "balance_change": "balance_change.normalized_value"
                                            },
                                            "script": """
                                              if (params.balance_change > 0) {return null;}
                                              else {
                                                return params.balance_change;
                                              }
                                            """
                                        }
                                    },
                                    "moving_average_of_burndown": {
                                        "moving_avg": {
                                            "buckets_path": "burndown",
                                            "gap_policy": "insert_zeros",
                                            "window": 7,
                                            "model": "ewma",
                                            "settings": {
                                                "alpha": 0.5
                                            }
                                        }
                                    },
                                    "second_order": {
                                        "derivative": {
                                            "buckets_path": "balance_change"
                                        }
                                    }
                                }
                            }
                        }
                    }
                }
            }
        }
    }

Here is a plot of my results


(Uy) #2

Apparently it has been discussed before:

If there are any other way of solving the problem than the workaround I would like to here it!


(system) #3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.