# How to search for anomalies in metrics?

**URL:** <https://discuss.elastic.co/t/how-to-search-for-anomalies-in-metrics/221953>\
**Category:** Elasticsearch\
**Created:** [March 3, 2020, 10:58pm UTC](https://discuss.elastic.co/t/how-to-search-for-anomalies-in-metrics/221953 "2020-03-03T22:58:49Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![jerrac](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jerrac/32/52980_2.png) [@jerrac](https://discuss.elastic.co/u/jerrac)\
**Post date:** [March 3, 2020, 10:58pm UTC](https://discuss.elastic.co/t/how-to-search-for-anomalies-in-metrics/221953/1 "2020-03-03T22:58:49Z")

</div>

I want to create a small script that will query ES for data, and then send me an alert when it sees that there is "anomalous" data.

For example, if the cpu load on a vm is suddenly spiking. I'd like an alert. That's easy enough to do if I just set a threshold, but I'd rather check for a sudden change in load/usage. Some vm's will naturally have a high cpu load, or RAM usage, with others will not.

I am digging into Elasticsearch Query DSL and the various aggregations to try and create my own script for this. Basically, run a query, check for a condition, and then send an alert, or not.

All my search results on this topic end up at proprietary solutions or Elastalert. I have no budget for this, and my attempts at getting Elastalert working were not successful. Though I may revisit once I understand how to search ES better.

Here are a few specific things I want to watch for:

A) If CPU load has gone up by more than 200% in the past 30 minutes.  
B) If RAM usage has gone up by more than 200% in the past 30 minutes.  
C) If the number of Apache requests has suddenly gone down by more than 50% in the past 60 minutes.

How would you go about watching for those things?

In my research, I discovered the [Median Absolute Deviation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-median-absolute-deviation-aggregation.html) aggregation. Would watching that be a way to get close to what I'm after?

I was able to build this query:

```auto
GET /_search
{
 "query": {
    "bool": {
      "must": [
        {
          "match_all": {}
        }
      ],
      "filter": [
        {
          "match_phrase": {
            "host.name": "learnescentos7"
          }
        },
        {
          "match_phrase": {
            "agent.type": "metricbeat"
          }
        },
        {
          "match_phrase": {
            "metricset.name": "load"
          }
        },
        {
          "range": {
            "@timestamp": {
              "gte": "now-30m",
              "lte": "now",
              "time_zone": "America/Los_Angeles"
            }
          }
        }
      ],
      "should": [],
      "must_not": []
    }
  },
  "aggs": {
    "avg_load_1": { "avg": { "field": "system.load.1" }},
    "max_load_1": { "max": { "field": "system.load.1" }},
    "min_load_1": { "min": { "field": "system.load.1" }},
    "variability_1": { "median_absolute_deviation": { "field": "system.load.1" }},
    "avg_load_5": { "avg": { "field": "system.load.5" }},
    "max_load_5": { "max": { "field": "system.load.5" }},
    "min_load_5": { "min": { "field": "system.load.5" }},
    "variability_5": { "median_absolute_deviation": { "field": "system.load.5" }},
    "avg_load_15": { "avg": { "field": "system.load.15" }},
    "max_load_15": { "max": { "field": "system.load.15" }},
    "min_load_15": { "min": { "field": "system.load.15" }},
    "variability_15": { "median_absolute_deviation": { "field": "system.load.15" }}
  }
}

```

Would alerting on variability\_1 being larger than 0 get me anywhere close to knowing if cpu load has gone up? I think I might be off track here since I think that aggregation will represent load going down as much as it does going up....

Any advice would be welcome. 🙂 Thanks!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 31, 2020, 10:59pm UTC](https://discuss.elastic.co/t/how-to-search-for-anomalies-in-metrics/221953/2 "2020-03-31T22:59:58Z")

</div>

Closed on March 31, 2020, 10:59pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.