Missing prediction values with moving average aggregation or moving function in Elasticsearch

[jmou]

Hello there, I'm getting 2 issues when trying to use the moving average aggregation functions in Elasticsearch to generate Holtwinters predictions:

• getting missing parts of prediction values (getting "null" instead) towards the end of the prediction period. I've also tried to visualize the the same piece of data on Grafana with the Holt-Winters prediction function, which succeeds in getting all the predictions with the current time duration of the actual data.

• "moving_avg" allows the option of "predict" to accept the number of days for the predictions into future beyond the existing time range of the actual data. Yet since "moving_avg" is going to be deprecated, I'm checking to use "MovingFunctions.holtWinters", and I don't find the input parameter of "predict" to accept the number of days to predict in future. I found out the documentation for a similar function "MovingFunctions.holtWintersForecast" which seems to accept the "predict" value from this link: https://artifacts.elastic.co/javadoc/org/elasticsearch/elasticsearch/6.4.0/org/elasticsearch/search/aggregations/pipeline/movfn/MovingFunctions.html#holtWintersForecast(double[],double,double,double,int,double,boolean,int). However, it reports exception of "illegal_argument_exception", with reason being "Unknown call [holtWintersForecast] with [8] arguments on type [org.elasticsearch.search.aggregations.pipeline.movfn.MovingFunctions]."

Any suggestions on how to fix the missing values, and how to enable the "moving_fn" to predict future values? I have attached the methods tried, and the screenshots of the results below. My version is Elasticsearch v6.4.2. Thanks!

data:
The data is time series data with independent variables "date" and "x", and dependent variable "y". It is stored into index "test-index". The time range of the actual data is from "2018-2-24" to "2018-8-16" for "x=a", with some missing data in it.

methods tried
I tried both the "moving_avg" that is said to be deprecated later, and also the "moving_fn" methods, and they both got the similar behaviors of lots of "null" values towards the end. I've also tried to replicate the query script from Grafana query inspector through Kibana - Dev Tools, which also gets missing data. The results from 3 methods are all sorted in descending order by date. Yet all their predictions are not available until the date of "2018-07-28".

1. "moving_avg".
   '''
   POST /test-index/_search?pretty
   {
   "size": 0,
   "query": {
   "bool": {
   "must": [
   {"term": {"x": "a"}}
   ]
   }
   },
   "aggs": {
   "date_aggs": {
   "date_histogram": {
   "field": "date",
   "interval": "1d",
   "format": "yyyy-MM-dd",
   "order": {
   "_key": "desc"
   }
   },
   "aggs": {
   "the_max": {
   "max": {"field": "y"}
   },
   "holtwinters_prediction": {
   "moving_avg": {
   "buckets_path": "the_max",
   "model": "holt_winters",
   "window": 21,
   "predict": 14,
   "settings": {
   "type": "add",
   "alpha": 0.8,
   "beta": 0,
   "gamma": 0,
   "period": 7
   }
   }
   }
   }
   }
   }
   }
   '''

2. "MovingFunctions.holtWinters".
   '''
   POST /test-index/_search?pretty
   {
   "size": 0,
   "query":{
   "bool":{
   "must": [
   {"term": {"x": "a"}}
   ]
   }
   },
   "aggs": {
   "date_aggs": {
   "date_histogram": {
   "field": "date",
   "interval": "1d",
   "format": "yyyy-MM-dd",
   "order": {
   "_key": "desc"
   }
   },
   "aggs": {
   "the_max": {
   "max": {"field": "y"}
   },
   "holtwinters_prediction": {
   "moving_fn": {
   "buckets_path": "the_max",
   "window": 21,
   "script": "if (values.length >= 7*2) {MovingFunctions.holtWinters(values, 0.8, 0, 0.3, 7, false)} else {Double.NaN}"
   }
   }
   }
   }
   }
   }
   '''
   

   
   
   
   

3. replication of Grafana script.
   '''
   POST /test-index/_search?pretty
   {
   "size":0,
   "query":{
   "bool":{
   "filter":[
   {"query_string":{
   "analyze_wildcard":true,
   "query":"x:a"
   }
   }
   ]
   }
   },
   "aggs":{
   "date_aggs":{
   "date_histogram":{
   "interval":"1d",
   "field":"date",
   "min_doc_count":0,
   "format":"yyyy-MM-dd",
   "order": {
   "_key": "desc"
   }
   },
   "aggs":{
   "the_max":{
   "max":{"field":"y"}
   },
   "holtwinters_prediction":{
   "moving_avg":{
   "buckets_path":"the_max",
   "minimize":false,
   "model":"holt_winters",
   "window":21,
   "predict":14,
   "settings":{
   "alpha":0.8,
   "beta":0,
   "pad":false,
   "period":7
   }
   }
   }
   }
   }
   }
   }
   '''
   

   
   
   
   

I also tried the "MovingFunctions.holtWintersForecast" to predict values into future, yet failed to get results returned.
'''
POST /test-index/_search?pretty
{
"size": 0,
"query":{
"bool":{
"must": [
{"term": {"x": "a"}}
]
}
},
"aggs": {
"date_aggs": {
"date_histogram": {
"field": "date",
"interval": "1d",
"order": {
"_key": "desc"
}
},
"aggs": {
"the_max": {
"max": {"field": "y"}
},
"holtwinters_prediction": {
"moving_fn": {
"buckets_path": "the_max",
"window": 21,
"script": "if (values.length >= 7*2) {MovingFunctions.holtWintersForecast(values, 0.8, 0, 0.3, 7, 0, true, 14)}"
}
}
}
}
}
}
'''

Meanwhile, this is the screenshot of Grafana's success in showing all the prediction data.