# How long do we have to retain the old data in X-Pack?

**URL:** <https://discuss.elastic.co/t/how-long-do-we-have-to-retain-the-old-data-in-x-pack/104832>\
**Category:** Elasticsearch\
**Created:** [October 23, 2017, 4:46am UTC](https://discuss.elastic.co/t/how-long-do-we-have-to-retain-the-old-data-in-x-pack/104832 "2017-10-23T04:46:09Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Arun\_Raj](https://avatars.discourse-cdn.com/v4/letter/a/958977/32.png) [@Arun\_Raj](https://discuss.elastic.co/u/Arun_Raj)\
**Post date:** [October 23, 2017, 4:46am UTC](https://discuss.elastic.co/t/how-long-do-we-have-to-retain-the-old-data-in-x-pack/104832/1 "2017-10-23T04:46:09Z")

</div>

Hi Everyone

I 'm new to X-pack ML while experimenting I came across a doubt - I ve feed one month data into X-pack for ML and the system would identified the pattern and calculated the probability, and it would able to find the anomalies in the next set of data as well. Can we delete the index from ES once the system identified the pattern? or So how long do we have to retain the old data in ES ? what will happen if I delete the old data from ES index ?

Thanks  
Arun

---

<div class="post-metadata">

**Author:** ![dkyle](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dkyle/32/59114_2.png) [@dkyle](https://discuss.elastic.co/u/dkyle)\
**Post date:** [October 23, 2017, 8:52am UTC](https://discuss.elastic.co/t/how-long-do-we-have-to-retain-the-old-data-in-x-pack/104832/2 "2017-10-23T08:52:46Z")

</div>

Hi Arun,

Yes you can delete old data after it has been analysed by ML but remember that you will not see that data in the UI when you look at historical anomalies.

ML has a setting `results_retention_days`, after this period has expired the old results are removed. You can [update the job](https://www.elastic.co/guide/en/elasticsearch/reference/5.6/ml-update-job.html) with this setting. I recommend thinking about how long you want to keep the old anomalies for (e.g. 30 days), setting `results_retention_days` to that period and use [Curator](https://github.com/elastic/curator) to prune the source indices at the same cadence.

---

<div class="post-metadata">

**Author:** ![Arun\_Raj](https://avatars.discourse-cdn.com/v4/letter/a/958977/32.png) [@Arun\_Raj](https://discuss.elastic.co/u/Arun_Raj)\
**Post date:** [October 23, 2017, 9:28am UTC](https://discuss.elastic.co/t/how-long-do-we-have-to-retain-the-old-data-in-x-pack/104832/3 "2017-10-23T09:28:23Z")

</div>

Thanks David.

So if I feed the new data into ES does the system will find anomalies with the old pattern ?

---

<div class="post-metadata">

**Author:** ![dkyle](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dkyle/32/59114_2.png) [@dkyle](https://discuss.elastic.co/u/dkyle)\
**Post date:** [October 23, 2017, 9:56am UTC](https://discuss.elastic.co/t/how-long-do-we-have-to-retain-the-old-data-in-x-pack/104832/4 "2017-10-23T09:56:21Z")

</div>

ML will continuously learn from new data. As your data changes ML will adjust it's understanding of what is anomalous. The 'pattern' isn't fixed it changes with the data.

---

<div class="post-metadata">

**Author:** ![Arun\_Raj](https://avatars.discourse-cdn.com/v4/letter/a/958977/32.png) [@Arun\_Raj](https://discuss.elastic.co/u/Arun_Raj)\
**Post date:** [October 23, 2017, 10:06am UTC](https://discuss.elastic.co/t/how-long-do-we-have-to-retain-the-old-data-in-x-pack/104832/5 "2017-10-23T10:06:35Z")

</div>

Thanks David for your clarification 🙂

---

<div class="post-metadata">

**Author:** ![Arun\_Raj](https://avatars.discourse-cdn.com/v4/letter/a/958977/32.png) [@Arun\_Raj](https://discuss.elastic.co/u/Arun_Raj)\
**Post date:** [October 24, 2017, 5:51am UTC](https://discuss.elastic.co/t/how-long-do-we-have-to-retain-the-old-data-in-x-pack/104832/6 "2017-10-24T05:51:45Z")

</div>

David, one more doubt - If we start and run a ML job(say the job will run for a month) how long do we have to retain the old data ?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 21, 2017, 5:51am UTC](https://discuss.elastic.co/t/how-long-do-we-have-to-retain-the-old-data-in-x-pack/104832/7 "2017-11-21T05:51:59Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.