How long do we have to retain the old data in X-Pack?

Elastic Stack Elasticsearch
1

Hi Everyone

I 'm new to X-pack ML while experimenting I came across a doubt - I ve feed one month data into X-pack for ML and the system would identified the pattern and calculated the probability, and it would able to find the anomalies in the next set of data as well. Can we delete the index from ES once the system identified the pattern? or So how long do we have to retain the old data in ES ? what will happen if I delete the old data from ES index ?

Thanks
Arun

2

Hi Arun,

Yes you can delete old data after it has been analysed by ML but remember that you will not see that data in the UI when you look at historical anomalies.

ML has a setting results_retention_days, after this period has expired the old results are removed. You can update the job with this setting. I recommend thinking about how long you want to keep the old anomalies for (e.g. 30 days), setting results_retention_days to that period and use Curator to prune the source indices at the same cadence.

3

Thanks David.

So if I feed the new data into ES does the system will find anomalies with the old pattern ?

4

ML will continuously learn from new data. As your data changes ML will adjust it's understanding of what is anomalous. The 'pattern' isn't fixed it changes with the data.

5

Thanks David for your clarification :slight_smile:

6

David, one more doubt - If we start and run a ML job(say the job will run for a month) how long do we have to retain the old data ?

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.