Is ML mainly for tracking anomalies?

[seanziee (Kofi)]

I'm getting started with Machine Learning with x-pack and it seems like the main capacity of it is to see anomalies, which I'm not in dire need of. I'm not sure if ML can help me with my use case so I'd appreciate if anyone has feedback for me.

I'm trying to correlate some errors based off of their timestamp. So if error 1 happens at 1,3,5 pm and error 99 happens at 1,3,5 pm also, I'd like to correlate the two and say that depending on how many times the two overlap in timestamp will determine how related they are; these two would be considered correlated. Can ML help me with that?

Thanks

[richcollier]

Hello,

No, sounds like you don't need ML for such a situation, since it is a static rule definition. You could accomplish this with a chain input in Watcher.

[seanziee (Kofi)]

@richcollier Oh cool! How would you use the chain input? I have 100's of error codes that I'm trying to correlate based off their timestamp to see which error codes in a given day are sending off at the same time.

[richcollier]

Oh, well, if you have hundreds of combinations of error codes, then ML might actually be what you want after all. (I took your example above as a one-off). However, before I go any further (and looking at the headline of this post) - you should indeed know that X-Pack ML is indeed mainly for tracking anomalies. It can detect unusually high occurrence rates of errors by type. If that's interesting, read on. Instead, if you need to alert on every occurrence of an error, then a static rule set that you create with Watches will be your best bet.

In ML, you could make a multi-metric job that was a "count" based detection, split by error code (here I am using weblogs and showing count by HTTP status code):

So, what you would get is a grid of anomalies by code:

Here I can see correlations between a spike in status=200 with spikes in status=304 and status=302.

Not sure if that's useful to you after all!

[seanziee (Kofi)]

Hmm yeah I see what you're describing but sadly I'm not as interested in anomalies, but rather seeing which error codes are sending in unison. Thanks though!