Anomaly detection on ratio of two counts

Elastic Stack Elasticsearch
1

Hi, I'm indexing a log file where the majority of the lines contain actions but a small percentage (1-2%) contains error lines.

I have managed to successfully plot the hourly error rate using either Timelion or the Visual Builder.

However, what I'd like to do is to perform anomaly detection on the hourly error rate.

What's the easiest way of doing this?

2

Create a bucket script aggregation: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline-bucket-script-aggregation.html

ML jobs can use aggregations as input. See: https://www.elastic.co/guide/en/elastic-stack-overview/current/ml-configuring-aggregation.html

3

@richcollier, thanks for the reply.
Is there a way to store the bucket script aggregation as a new index?
Or is this just for the purposes of the ML job?
In other words, wonder if this script can be stored generically and then used in plots, ML, watcher, etc.?

4

If you want to use that info for things other than ML, you could possibly build a summarization job in Watcher that will routinely query, aggregate and index the output.

Otherwise, if the information is only for the purposes of ML, then just do the aggregations in-line within the ML job

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

6

Prior advice of using Watcher for a summarization job is now irrelevant given the addition of the Transform capability.