How to index anomalous documents in another index?

Elastic Stack Elasticsearch
1

It is my first time working with Elasticsearch anomaly detection. I've read the concepts and create a sample index to test it. I've also created these job and datafeed by using kibana for the sample index:

{
  "job_id": "account",
  "description": "",
  "groups": [],
  "analysis_config": {
    "bucket_span": "1m",
    "detectors": [
      {
        "function": "high_mean",
        "field_name": "credit"
      }
    ],
    "influencers": [],
    "summary_count_field_name": "doc_count"
  },
  "data_description": {
    "time_field": "@timestamp"
  },
  "custom_settings": {
    "created_by": "single-metric-wizard"
  },
  "analysis_limits": {
    "model_memory_limit": "11MB"
  },
  "model_plot_config": {
    "enabled": true,
    "annotations_enabled": true
  }
}

{
  "datafeed_id": "datafeed-account",
  "job_id": "account",
  "indices": [
    "account"
  ],
  "query": {
    "bool": {
      "must": [
        {
          "match_all": {}
        }
      ]
    }
  },
  "runtime_mappings": {},
  "aggregations": {
    "buckets": {
      "date_histogram": {
        "field": "timeStamp",
        "interval": "6000ms"
      },
      "aggregations": {
        "credit": {
          "avg": {
            "field": "credit"
          }
        },
        "@timestamp": {
          "max": {
            "field": "timeStamp"
          }
        }
      }
    }
  }
}

I need to index the anomalous documents in another index, so it must be possible to define something like action concept (e.g. alert, index, ...) in the job but I don't know how to define it.
How can I index the anomalous events in another index?

2

Anomaly Detection jobs create results into a new index already (and are named .ml-anomalies-*). Using the Machine Learning UI in Kibana will read the results from these indices and show them to you but obviously, you're also able to manually query those indices for other purposes (alerting, etc.)

If you want to learn the basics of Elastic ML, download an e-copy of this book:
https://events.elastic.co/machinelearningwithelastic

and/or watch the webinar: “Machine Learning with the Elastic Stack” book preview - Part 1: Anomaly Detection | Elastic Videos

1 Like
3

Thank you Rich,
It's also looks like possible to send results to dedicated indices.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.