# Insufficient History for prediction

**URL:** <https://discuss.elastic.co/t/insufficient-history-for-prediction/241711>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-machine-learning\
**Created:** [July 17, 2020, 7:07pm UTC](https://discuss.elastic.co/t/insufficient-history-for-prediction/241711 "2020-07-17T19:07:03Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Shubhangi](https://avatars.discourse-cdn.com/v4/letter/s/9de053/32.png) [@Shubhangi](https://discuss.elastic.co/u/Shubhangi)\
**Post date:** [July 17, 2020, 7:07pm UTC](https://discuss.elastic.co/t/insufficient-history-for-prediction/241711/1 "2020-07-17T19:07:03Z")

</div>

Hi,  
I'm using Elastic ML 7.8.

I've a monthly index with 350k+ documents count/events per month or 118.3mb worth of data per month. I've data from March to July present (4 months+ data). So, in total, that is 1400k events with 473.2mb worth of data, still I'm getting the following error when I'm trying to run Forecast for a day.

> Detector[0]: Insufficient history to forecast

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/a/9a8b69001b17528e6bc64cc36abd74dbc505d2f7.png)

May I know how much data is sufficient data for ML forecasts? Please respond with the amount of data(GB or events).

Thanks!

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [July 17, 2020, 8:12pm UTC](https://discuss.elastic.co/t/insufficient-history-for-prediction/241711/2 "2020-07-17T20:12:15Z")

</div>

The general rule-of-thumb is that you should not ask for a forecast with a duration that exceeds your historical data. In other words, if I had a week's worth of data, I should not expect to be able to forecast out a month.

So, I need to ask a few questions:

1. What does the general trend of the data look like? Perhaps a screenshot of the data in the Single Metric Viewer
2. What detector function are you using?
3. What is the bucket span of the job?

---

<div class="post-metadata">

**Author:** ![Shubhangi](https://avatars.discourse-cdn.com/v4/letter/s/9de053/32.png) [@Shubhangi](https://discuss.elastic.co/u/Shubhangi)\
**Post date:** [July 18, 2020, 5:17pm UTC](https://discuss.elastic.co/t/insufficient-history-for-prediction/241711/3 "2020-07-18T17:17:53Z")

</div>

Thanks for replying, Rich.  
Here's the general trend of data:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/b/bbf728ddc8f1a2dd4b51acc82621d1279b9f1dfb.png)

Detector: high\_count() by lbname  
Bucket span: 3h

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [July 18, 2020, 5:53pm UTC](https://discuss.elastic.co/t/insufficient-history-for-prediction/241711/4 "2020-07-18T17:53:29Z")

</div>

1. since you are asking it to predict for every `lbname` I wonder if there are some instances of `lbname` that don't have very much data (although it looks like this one does). There may be a situation in which even 1 by\_field out of all of them could maybe cause this error to be thrown. How many `lbname`s are there and is there a reasonable way to check to see if every one has sufficient historical data?

1. I see that your data doesn't really have any discernable trends to it (i.e. no cyclicality, no general upwards slope, etc.). Even if you get this to work you might be underwhelmed by the forecast. You'll likely get a horizontal line at around the middle of the range of the data (somwhere around 30-ish in this case). Maybe it is not obvious, but the forcast capability extrapolates trends in the data - it doesn't "predict" anomalies. Anomalies are surprise events, by definition.

One thing you could try is to isolate the data for one `lbname` (filter in Discover tab and save it as a saved search). Then, create an ML job from that saved search on that single time series and do the forecast on that.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 15, 2020, 5:53pm UTC](https://discuss.elastic.co/t/insufficient-history-for-prediction/241711/5 "2020-08-15T17:53:37Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.