# Need help setting up some forecasting monitoring - Math guys, let's do this

**URL:** <https://discuss.elastic.co/t/need-help-setting-up-some-forecasting-monitoring-math-guys-lets-do-this/77169>\
**Category:** Kibana\
**Created:** [March 2, 2017, 4:06pm UTC](https://discuss.elastic.co/t/need-help-setting-up-some-forecasting-monitoring-math-guys-lets-do-this/77169 "2017-03-02T16:06:47Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![t.Farestad](https://avatars.discourse-cdn.com/v4/letter/t/3ab097/32.png) [@t.Farestad](https://discuss.elastic.co/u/t.Farestad)\
**Post date:** [March 2, 2017, 4:06pm UTC](https://discuss.elastic.co/t/need-help-setting-up-some-forecasting-monitoring-math-guys-lets-do-this/77169/1 "2017-03-02T16:06:47Z")

</div>

So for my particular need, I am attempting to make interval forecasting based upon counts. IE, we look through the logs for a certain LogEntryPhrase, that if exists, signifies that a certain popup was successful. I want to know how many popups occurred in the last 15 minutes. And then compare that to the same interval one week ago, two weeks ago, etc.

So for Monday from 9:00 - 9:15, how does that compare to last Monday, and the Monday before that, etc. Think call center forecasting based upon counts. Unfortunately, Kibana doesn't have this capability with date histograms, which is crucial for a lot of the built in smoothing / prediction methods (ewma/holt\_winters) since my interval needs to be specific time frame, on various days.

IE counts from 9:00 - 9:15 arn't very indicative of the expected counts from 9:15 - 9:30, but the 9:15-9:30 from the last few Mondays, is.

Below is my current hack method of doing it, but again, I will miss out on a ton of the smoothing techniques necessary. (How do I handle missing data / anomalies)

Any advice, or workarounds that someone has found that works for them?

{  
"size": 0,  
"query": {  
"bool": {  
"filter": {  
"range": {  
"@timestamp": {  
"gte": "now-10w"  
}  
}  
},  
"must": [Phrases I'm looking for are here]  
}  
},  
"aggs": {  
"history": {  
"date\_range": {  
"field": "@timestamp",  
"ranges": [  
{  
"from": "now-5w-15m",  
"to": "now-5w"  
},  
{  
"from": "now-4w-15m",  
"to": "now-4w"  
},  
{  
"from": "now-3w-15m",  
"to": "now-3w"  
},  
{  
"from": "now-2w-15m",  
"to": "now-2w"  
},  
{  
"from": "now-1w-15m",  
"to": "now-1w"  
}  
]  
},  
"aggs": {  
"my\_count": {  
"sum": {  
"script": "1"  
}  
}  
}  
},  
"my\_stats": {  
"extended\_stats\_bucket": {  
"buckets\_path": "history\>my\_count"  
}  
},  
"today": {  
"date\_range": {  
"field": "@timestamp",  
"ranges": [  
{  
"from": "now-15m",  
"to": "now"  
}  
]  
}  
}  
}  
}

Which gives me the output, I need. It's also incredibly tedious once we increase storage

For example: Results  
"Todays"  
"aggregations": {  
"today": {  
"buckets": [  
{  
"key": "2017-03-02T15:10:38.439Z-2017-03-02T15:25:38.439Z",  
"from": 1488467438439,  
"from\_as\_string": "2017-03-02T15:10:38.439Z",  
"to": 1488468338439,  
"to\_as\_string": "2017-03-02T15:25:38.439Z",  
"doc\_count": 23  
}  
]  
}

And then we get a few buckets that look like this, with their aggregated stats below:  
{  
"key": "2017-02-23T15:10:38.439Z-2017-02-23T15:25:38.439Z",  
"from": 1487862638439,  
"from\_as\_string": "2017-02-23T15:10:38.439Z",  
"to": 1487863538439,  
"to\_as\_string": "2017-02-23T15:25:38.439Z",  
"doc\_count": 28,  
"my\_count": {  
"value": 28  
}  
}  
]  
},  
"my\_stats": {  
"count": 5,  
"min": 19,  
"max": 28,  
"avg": 24.4,  
"sum": 122,  
"sum\_of\_squares": 3023,  
"variance": 9.239999999999963,  
"std\_deviation": 3.039736830714127,  
"std\_deviation\_bounds": {  
"upper": 24.4,  
"lower": 24.4  
}  
}

But again, I really am missing out on smoothing techniques that handle missing/anomalies. Any ideas?

---

<div class="post-metadata">

**Author:** ![thomasneirynck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thomasneirynck/32/23313_2.png) [@thomasneirynck](https://discuss.elastic.co/u/thomasneirynck)\
**Post date:** [March 2, 2017, 8:02pm UTC](https://discuss.elastic.co/t/need-help-setting-up-some-forecasting-monitoring-math-guys-lets-do-this/77169/2 "2017-03-02T20:02:12Z")

</div>

This is a tough one.

I'm not sure about smoothing methods, or how you would even integrate these in Kibana. Kibana will render your ES-results, but doesn't really do post-processing on the data. So you might try your luck in the ES-forum with your question [https://discuss.elastic.co/c/elasticsearch](https://discuss.elastic.co/c/elasticsearch).

a few of the cuff thoughts, wrt. smoothing:

are you talking about line-fitting, or interpolation?

- For line-fitting ([https://en.wikipedia.org/wiki/Curve\_fitting](https://en.wikipedia.org/wiki/Curve_fitting)), you'll be looking at regression methods, machine learning techniques. There is no easy answer there.
- For interpolation, you might get a long way with more basic geometric interpolations, most likely polynomial splines ([https://en.wikipedia.org/wiki/Spline\_(mathematics)](https://en.wikipedia.org/wiki/Spline_(mathematics)). Those work reasonably well for time-series data.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 30, 2017, 8:02pm UTC](https://discuss.elastic.co/t/need-help-setting-up-some-forecasting-monitoring-math-guys-lets-do-this/77169/3 "2017-03-30T20:02:12Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.