ML datafeed bucket aggregation/script challenge

Elastic Stack Elasticsearch
1

I would like to create an anomaly detection job for the ratio of two counts. The logic is as follows...

A = count filtered by condition X
B = count filtered by condition Y
C = A/B

The datafeed would need to produce C as a field to be used in the analysis_config.

I have been trying to figure out some combination of bucket aggregation/bucket script/scripted fields that would work as a datafeed. The first challenge is whether it is even possible to have any query that produces two bucket aggregation values each derived from a different filter condition? If I can get these two values a bucket_script would be easy enough to do the math.

Does anyone have any tips on how something like this can be done?

2

Hi Rob - this should help: Analyzing a ratio of documents over time with Anomaly Detection

3

@richcollier that was a big help. Thanks!

1 Like
4

Is there any way to get a field into the aggregation-based datafeed that can be used for as a partition_field? It is possible to add levels of aggregation that create the buckets. However I haven't been able to get the key for the buckets as a field that can be accessed in the detectors config.

5

Robert, here's an example using the demo dataset farequote:

(note I put a terms agg of size 100 but there are only like 19 different airlines)

PUT _ml/anomaly_detectors/farequote_terms_agg
{
  "analysis_config": {
    "bucket_span": "5m",
    "detectors": [{
      "function": "mean",
      "field_name": "responsetime",  
      "partition_field_name": "airline"  
    }],
   "influencers" : ["airline"],
    "summary_count_field_name": "doc_count"
  },
  "data_description": {
    "time_field":"@timestamp"  
  },
  "datafeed_config":{
    "indices": ["farequote"],
    "aggregations": {
      "buckets": {
        "date_histogram": {
          "field": "@timestamp",
          "fixed_interval": "5m",
          "time_zone": "UTC"
        },
        "aggregations": {
          "@timestamp": {  
            "max": {"field": "@timestamp"}
          },
          "airline": {  
            "terms": {
             "field": "airline",
              "size": 100
            },
            "aggregations": {
              "responsetime": {  
                "avg": {
                  "field": "responsetime"
                }
              }
            }
          }
        }
      }
    }
  }
}
6

That worked. The location in the hierarchy of aggregations is important. The key here was that it needs to be at the same level as @timestamp, i.e. "inside" the date histogram.

7

Here is the final result:

image
image1422×1004 114 KB

1 Like
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.