Retraining ML jobs

[seanziee (Kofi)]

I'm currently using ML to track anomalies in my system. I'm wondering if it's possible to mark certain anomalies as incorrect results and have ML use that in its future predictions? I saw the rule functionality but that seems to be more like creating a threshold for certain types of events and filtering those out in the future, rather than retraining.

Thanks!

[darnautov (Dima Arnautov)]

Hi @seanziee,

The concept of unsupervised learning and anomaly detection implies you do not label your data manually. You can utilize model snapshots, so in case it's a one-off anomaly, you can revert the snapshot. Also if you know about such events in advance you can create a scheduled event in the calendar. Hope it helps.

Regards,
Dima

[Tom_Veasey]

I will also add that, although Dima covers all functionality built in for this, we are actively considering something closer to what you request.

There are two main ways we might proceed. With supervised learning features in the stack we can train, for example, a classifier of important (groups of) anomalies based on user feedback. (At a push, this is something one can develop already independently: for example see this excellent talk by Bell.) In the nearer term, we may provide a sensitivity control which allows one to adjust the tradeoff between FP and FN for anomaly detection in a less granular fashion.

There are some hard aspects of getting a supervised approach to work for this problem if you want to make a very shrink wrapped user experience:

1. You may not get a lot of rather imbalanced training data,
2. You need to find features which will generally be effective at learning the right things from user feedback,
3. You want retraining with small amounts of additional data to be stable so adding a small amount of feedback doesn't significantly alter predictions.

Regarding point 3, we are currently working on incremental training which addresses this for our stack based supervised ML. When this work is nearer to completion we will likely revisit creating a more streamlined process for users feeding back on specific anomalies.

[seanziee (Kofi)]

Thanks to both of your for the detailed response. Thanks for the tool and I'm happy to be using it.