# Anomaly detection on ratio of two counts

**URL:** <https://discuss.elastic.co/t/anomaly-detection-on-ratio-of-two-counts/164842>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-machine-learning\
**Created:** [January 18, 2019, 6:17pm UTC](https://discuss.elastic.co/t/anomaly-detection-on-ratio-of-two-counts/164842 "2019-01-18T18:17:07Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![nandrik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nandrik/32/39951_2.png) [@nandrik](https://discuss.elastic.co/u/nandrik)\
**Post date:** [January 18, 2019, 6:17pm UTC](https://discuss.elastic.co/t/anomaly-detection-on-ratio-of-two-counts/164842/1 "2019-01-18T18:17:07Z")

</div>

Hi, I'm indexing a log file where the majority of the lines contain actions but a small percentage (1-2%) contains error lines.

I have managed to successfully plot the **hourly error rate** using either Timelion or the Visual Builder.

However, what I'd like to do is to perform **anomaly detection** on the **hourly error rate**.

_What's the easiest way of doing this?_

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [January 22, 2019, 2:53pm UTC](https://discuss.elastic.co/t/anomaly-detection-on-ratio-of-two-counts/164842/2 "2019-01-22T14:53:33Z")

</div>

Create a bucket script aggregation: [https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline-bucket-script-aggregation.html](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-pipeline-bucket-script-aggregation.html)

ML jobs can use aggregations as input. See: [https://www.elastic.co/guide/en/elastic-stack-overview/current/ml-configuring-aggregation.html](https://www.elastic.co/guide/en/elastic-stack-overview/current/ml-configuring-aggregation.html)

---

<div class="post-metadata">

**Author:** ![nandrik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nandrik/32/39951_2.png) [@nandrik](https://discuss.elastic.co/u/nandrik)\
**Post date:** [January 22, 2019, 3:35pm UTC](https://discuss.elastic.co/t/anomaly-detection-on-ratio-of-two-counts/164842/3 "2019-01-22T15:35:38Z")

</div>

@richcollier, thanks for the reply.  
Is there a way to store the bucket script aggregation as a new index?  
Or is this just for the purposes of the ML job?  
In other words, wonder if this script can be stored generically and then used in plots, ML, watcher, etc.?

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [January 22, 2019, 3:39pm UTC](https://discuss.elastic.co/t/anomaly-detection-on-ratio-of-two-counts/164842/4 "2019-01-22T15:39:28Z")

</div>

If you want to use that info for things other than ML, you could possibly build a summarization job in Watcher that will routinely query, aggregate and [index the output](https://www.elastic.co/guide/en/elastic-stack-overview/current/actions-index.html).

Otherwise, if the information is only for the purposes of ML, then just do the aggregations in-line within the ML job

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 19, 2019, 3:39pm UTC](https://discuss.elastic.co/t/anomaly-detection-on-ratio-of-two-counts/164842/5 "2019-02-19T15:39:30Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [March 26, 2020, 1:26pm UTC](https://discuss.elastic.co/t/anomaly-detection-on-ratio-of-two-counts/164842/6 "2020-03-26T13:26:57Z")

</div>

Prior advice of using Watcher for a summarization job is now irrelevant given the addition of the [Transform](https://www.elastic.co/guide/en/elasticsearch/reference/current/transforms.html) capability.