# \[6.1.1\] Machine Learning Job: reanalyse datapoints / ignore last bucket

**URL:** <https://discuss.elastic.co/t/6-1-1-machine-learning-job-reanalyse-datapoints-ignore-last-bucket/149159>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-machine-learning\
**Created:** [September 19, 2018, 4:59pm UTC](https://discuss.elastic.co/t/6-1-1-machine-learning-job-reanalyse-datapoints-ignore-last-bucket/149159 "2018-09-19T16:59:56Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![gmatheus](https://avatars.discourse-cdn.com/v4/letter/g/bb73d2/32.png) [@gmatheus](https://discuss.elastic.co/u/gmatheus)\
**Post date:** [September 19, 2018, 4:59pm UTC](https://discuss.elastic.co/t/6-1-1-machine-learning-job-reanalyse-datapoints-ignore-last-bucket/149159/1 "2018-09-19T16:59:56Z")

</div>

Hi!

I'm trying to configure my ML Job + Watcher in a specific ingestion scenario:

- Ingestions occurs once in a day and the events timestamps are aggregated in the beginning of the day.
- The daily ingestion updates the day before yesterday ( **now-2d/d** ) values, but also inserts some documents in yesterday bucket ( **now-1d/d** ) - that will be replaced in the next ingestion (tomorrow).
- The daily ingestion also update old buckets (e.g.: last week buckets).

Rules:

- It should ignore last bucket ( **now-1d/d** ) since this value will be replaced tomorrow;
- It should reanalyse old buckets updates (e.g: last week buckets).
- The watcher should target the day before yesterday ( **now-2d/d** ) with the correct value analysis (I think this can be accomplished with a `range` + `gte` & `lte` filter in watcher)

I tested several `delay` values for datafeed but it seems that the job always fetch data from last bucket before it updates to the correct value (that's why I need to ignore last datapoint):  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/f/7f707ecdaf6a8a725cc3495a771610a5996343ac.png)

Is it possible to configure a Job with these rules above?

Thanks!

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [September 20, 2018, 1:22pm UTC](https://discuss.elastic.co/t/6-1-1-machine-learning-job-reanalyse-datapoints-ignore-last-bucket/149159/2 "2018-09-20T13:22:57Z")

</div>

For the most part, the "normal" operation of ML is to analyze data in chronological order without any attempt to "go back in time" and re-analyze older data. The setting of `query_delay` is meant to simply delay analysis to account for delays in ingest.

The only way to have an ML job re-analyze old buckets is to simply have "disposable" ML jobs that essentially analyze all of the historical data every single day.

This could be automated via daily scripts that use the [ML API](https://www.elastic.co/guide/en/x-pack/current/ml-api-quickref.html) to create, run, and report on a job.

So, in effect:

1. create the daily job
2. run this job over all historical data that's relevant to you
3. report/alert as necessary
4. delete daily job
5. start the entire process over tomorrow

---

<div class="post-metadata">

**Author:** ![gmatheus](https://avatars.discourse-cdn.com/v4/letter/g/bb73d2/32.png) [@gmatheus](https://discuss.elastic.co/u/gmatheus)\
**Post date:** [September 20, 2018, 9:49pm UTC](https://discuss.elastic.co/t/6-1-1-machine-learning-job-reanalyse-datapoints-ignore-last-bucket/149159/3 "2018-09-20T21:49:14Z")

</div>

Hey @richcollier, thanks for your help!

Besides the docs itself, do you have a real example of the ML API usage? Can be a blog post, forum discussion, anything that could help me to start this out.

Thanks!

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [September 21, 2018, 1:01pm UTC](https://discuss.elastic.co/t/6-1-1-machine-learning-job-reanalyse-datapoints-ignore-last-bucket/149159/4 "2018-09-21T13:01:56Z")

</div>

Take a look here for an example: [https://gist.github.com/richcollier/5482702c7bef6de9a14ff29fa39ef21a](https://gist.github.com/richcollier/5482702c7bef6de9a14ff29fa39ef21a)

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [October 29, 2018, 10:04pm UTC](https://discuss.elastic.co/t/6-1-1-machine-learning-job-reanalyse-datapoints-ignore-last-bucket/149159/5 "2018-10-29T22:04:24Z")

</div>