ML newbie - where are results stored and for how long

[stefws (Steffen Winther Sørensen)]

Are initially playing a bit with ML and wondering about where and for how long ML processing results are stored and how to maintain such data.

Eg. got a ML job which seems to indicate anormalities prior to existing data as data are removed by index management. How to like-wise maintain ML results?

[edsavage]

Hi Steffen,

thanks for playing with our ML product. We hope you're enjoying it so far.

The results are stored in a special "system" index. By default this index is called .ml-anomalies-shared. You can view this index from the management console but you will need to Include system indices to view them:

You can get some brief information about your results by clicking on the index name and then clicking on the Summary tab.

more detailed information can be found by exploring the other tabs.

How long the results are stored is configurable. Full details are in the documentation https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-job-resource.html. The relevant setting is

results_retention_days

(long) Advanced configuration option. The number of days for which job results are retained. Once per day at 00:30 (server time), results older than this period are deleted from Elasticsearch. The default value is null, which means results are retained.

Hope this helps & happy festive season!

Ed

[stefws (Steffen Winther Sørensen)]

Thanks for your reply!

Could the advanced settings be edited for an existing job?

I only found it possible to do this by cloning an existing job and then adding a setting in the 'Edit JSON' tab

[edsavage]

Morning Steffen,

Cloning the job in order to reconfigure is indeed a valid approach but is inconvenient for long running or real time jobs.

We do offer a REST API that allows updating the configuration of certain parameters and luckily results_retention_days is on that list https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-update-job.html

You can use the Dev Tools console to interact with the REST API

or, if you prefer you can always use 3rd party utilities such as curl, postman etc.

Hope that helps,

Cheers, Ed

[stefws (Steffen Winther Sørensen)]

@edsavage Thanks!

[edsavage]

np