ML/Alerting for amount of successful transactions

Elastic Observability
,
1

Hello,

I would like to create some sort of alert that uses the ML capability to notify me if the successful transaction rate for a specific service is anomalous.

For example:
Let's assume the trend of successful transactions is 100/day. If one day I get 50 or 150 I get notified and I can check what's going on.

I have read the documentation and I understand I need to use the Anomaly Detection feature, but the pre-configured jobs all point to a specific property to monitor (e.g. service.name). I want to monitor the number of successful transactions for a specific service.

Could you please point me in the right direction for this?

Thanks in advance :slight_smile:

2

Create a job from the ML UI that points to the index of interest (or leverages a Saved Search of interest) and then uses the count detector:

image
image1414×1532 127 KB

Optionally, you can "split" the job "for every" categorical field (like service.name in your case, but here I will choose response.keyword

image
image2272×2146 411 KB

The end result is anomalies found in the spike or drop of events per unit time:

image
image2430×2094 370 KB

3

Thanks for your reply @richcollier.
I managed to create the ML job as you described, how can I turn it into some sort of alert? My end goal is to get notified if the number of successful transactions of the previous day was below average.

Thanks in advance.

4

If you are using 7.11 or earlier, ML uses Watcher to do the alerting (see old, but still relevant blog here: Alerting on Machine Learning Jobs in Elasticsearch | Elastic Blog)

If you are using 7.12+ ML now uses the new alerting framework in Kibana. See the new docs here: Configuring anomaly detection alerts | Machine Learning in the Elastic Stack [7.12] | Elastic

1 Like
5

Thanks a lot. Is it my understanding correct that if I have a baseline of 100 requests per day, the alert will notify me if one day I get less or more? Or does it work only if I get more?

6

a detector configuration of the count detector function finds anomalies on both the high side and low side (whereas the one-sided detector functions high_count and low_count only detect anomalies in one direction).

A real astute user might notice that the highest sensitivity possible is two use a job that has two detectors configured (one with high_count and one with low_count). This configuration has higher fidelity, especially in the case where the dynamic ranges of anomalies on the high side are drastically different than on the low side (i.e. spikes that tend to be much larger in magnitude than the dips). Because each detector maintains its own normalization table, having separate normalized scoring for the two detectors gives more sensitivity control.

1 Like