How to craft watches for delta change anomalies?

Elastic Stack Elasticsearch
1

I think this has been asked before by somebody but without answers. Basically the question is how to define a watch that can detect delta change anomalies in indices. Say, the frequency of document additions changes from 10doc/s to 30doc/s (a spike). An naive solution would require 2 calls and save states from the first call for computation in the second call, and this may not be feasible in the context of watch execution.

I wonder if there is a better way out?

Thanks a lot!

Fang

1 Like
2

Just found out there was this date based aggregation. Can we use this to implement change detections. For example, breaking up the past 5 minutes into 5 buckets, and if the latest one is significantly larger or smaller than the avg of the last 4 (or something else), then raise a spike alert?

3

Hi Fang,

Yes, that's exactly what I was going to suggest :slight_smile:

In your query, you could do a date aggregation, and use a scripted condition to iterate over the results and decide whether the actions should be run.

In ES 2.0 / Watcher 2.0, we have a few new aggregations that will make this easier. One of these new aggregations, Derivatives, would make it easy to calculate the change between the date histogram buckets, so your script would just need to check the derivative value and see if it were above/below your change threshold.

In the Watcher docs, we show how to use scripted conditions. If you want to see an example, the Marvel memory usage watch shows a sophisticated way to use scripting in the condition.

Hope that helps!
Cheers,
Steve

4

Cool. I coded one watch with hourly buckets for a specific date and seems I can get the histogram counts properly. Now I have issues referencing metrics within each buckets using mustache syntax. the following line will generate empty value when the buckets variable is referred to.

"actions": {
"log": {
"logging": {
"text": "There were {{ctx.payload.aggregations.docs_over_time.buckets[-1].doc_count}} docs at {{ctx.execution_time}}"
}
}
}

any pointers on how to use mustache syntax to refer to json object values?

5

Can you share an example of what you want the log message to look like?

It would also be helpful to see the rest of the Watch definition, at least the query section, so we can see what structure to expect in the results.

6

Ok, I list the input section as follows.
"input": {
"search": {
"request": {
"indices": [
"sample_index_*"
],
"body": {
"query": {
"filtered": {
"query": {
"bool": {
"must": {
"match_phrase_prefix": {
"id": "21"
}
}
}
},

            "filter": {
              "bool": {
                "must": {
                    "range": { "time": { "lte": "now-3d", "gte": "now-4d" }}
                 }
              } 
            }
          }
        },
        "aggs" : {
              "docs_over_time" : {
                  "date_histogram" : {
                      "field" : "time",
                      "interval" : "1h"
                  }
              }
        }
      }
    }
  }

},

So the response includes a bunch of buckets in the aggregation with doc_count metric in each. The output should print the actual number of docs in the referred bucket. But I only got the following:

"There were docs at 2015-08-20T21:20:54.083Z"

7

The mustache syntax doesn't work quite the same as the groovy syntax you use in the condition. What I would do here would be to add a [transform stage][1] to your action and use groovy to identify the value you want and store it in the context, so you can easily reference the value from the mustache template.

If you always know which array item you want to reference, I believe this is proper mustache syntax for referring to the 4th item in the array:
ctx.payload.aggregations.docs_over_time.buckets.4.doc_count

Hope that helps!
[1]: https://www.elastic.co/guide/en/watcher/current/transform.html

8

Thanks. I realized that mustache had a different syntax but did not find any useful pointers, but I actually found out ctx.vars can be used to achieve the purposes.

watcher is indeed a very powerful feature, the only concern now is that the current python binding is very primitive. Is there any plan on a dsl based watcher lib in python any time soon?

Thanks a lot.

9

Great, I'm glad you sorted it out :slight_smile:

If you want to quickly iterate on a watch, I've found that the execute API is a good way to run a watch and see how it would have worked.

The current Watcher python client library is here: https://github.com/elastic/elasticsearch-watcher-py

10

There isn't much in the watcher python client library you mentioned. I was looking for something similar to elasticsearch-dsl-py, but also works for watcher apis.

11

Hi Fang,

If you are open to a 3rd party (commercial) plugin for the entire Elastic stack... My company, Prelert, makes this plugin that specifically does anomaly detection (download a free trial here).

Give it a try!

Cheers,

-Rich

13

Just putting this here for future viewers to see that advanced machine learning-based anomaly detection was introduced in v5.4 and went GA in v5.5:

https://www.elastic.co/products/x-pack/machine-learning