# How to get all relevant data of anomaly into alert message

**URL:** <https://discuss.elastic.co/t/how-to-get-all-relevant-data-of-anomaly-into-alert-message/130489>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-machine-learning\
**Created:** [May 3, 2018, 3:18pm UTC](https://discuss.elastic.co/t/how-to-get-all-relevant-data-of-anomaly-into-alert-message/130489 "2018-05-03T15:18:01Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![admlko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/admlko/32/21787_2.png) [@admlko](https://discuss.elastic.co/u/admlko)\
**Post date:** [May 3, 2018, 3:18pm UTC](https://discuss.elastic.co/t/how-to-get-all-relevant-data-of-anomaly-into-alert-message/130489/1 "2018-05-03T15:18:01Z")

</div>

I am trying to write a python script in order to find anomalies and relay them forward into our monitoring system.

What I am looking for is to get all the same information I can find from anomaly explorer (in the picture):

 ![Untitled](https://us1.discourse-cdn.com/elastic/original/3X/3/e/3e4575f6d0b8c9021a80d3a3e6522193c9ca2781.png)

But after a couple of days of trying, I just cannot get it right.

What I have done, is that I have a single ML job which uses "customer"-field for partitioning the data. The function is "high\_count by keywords over username partitionfield=customer".

If I have understood correctly, I should first search buckets, which has anomaly\_score greater of equal to 75 (critical), which would give me a timeframe when at least one anomaly happened.  
Then I would query all records and influencers from that timeframe, and I would get anomalies to be sent forward.

But my problem is that I don't know how to partition the data properly, as the bucket doesn't seem to have the information which customers data caused the anomaly. If I have understood correctly, it the bucket only tells the timeframe. So if I would query the influencers from that timeframe, I would also get other customers influencers and the data would get mixed. It would be trivial if every customer has their own ML job with separate indices, but I would like to have a single job for this thing.

Somehow the anomaly explorer gets it right. Can someone explain to me how it is done there?

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [May 3, 2018, 7:05pm UTC](https://discuss.elastic.co/t/how-to-get-all-relevant-data-of-anomaly-into-alert-message/130489/2 "2018-05-03T19:05:12Z")

</div>

In the .ml-anomalies-\* index, you need to query for `result_type:record` in order to get the detail that you see in the screenshot

---

<div class="post-metadata">

**Author:** ![admlko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/admlko/32/21787_2.png) [@admlko](https://discuss.elastic.co/u/admlko)\
**Post date:** [May 3, 2018, 8:37pm UTC](https://discuss.elastic.co/t/how-to-get-all-relevant-data-of-anomaly-into-alert-message/130489/3 "2018-05-03T20:37:03Z")

</div>

Thank you for your reply!

I thought that I can get the anomaly score from the bucket only, so can you elaborate a bit how I could calculate it myself using single records?

So the buckets are only to make querying faster (it functions as aggregated results from single records?), or is there another usage?

If single records contains influencers too, what is the function to have them also separately?

Thank you for taking the time to open the logic behind machine learning!

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [May 3, 2018, 8:49pm UTC](https://discuss.elastic.co/t/how-to-get-all-relevant-data-of-anomaly-into-alert-message/130489/4 "2018-05-03T20:49:19Z")

</div>

Take a look at this blog - it will explain a lot:

> **[Machine Learning Anomaly Scoring and Elasticsearch - How it Works
	  	 | Elastic](https://www.elastic.co/blog/machine-learning-anomaly-scoring-elasticsearch-how-it-works)**
>
> We often get questions about Elastic’s Machine Learning “anomaly score” and how the various scores presented in the dashboards relate to the “unusualness” of individual occurrences within the da...

---

<div class="post-metadata">

**Author:** ![admlko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/admlko/32/21787_2.png) [@admlko](https://discuss.elastic.co/u/admlko)\
**Post date:** [May 7, 2018, 6:04am UTC](https://discuss.elastic.co/t/how-to-get-all-relevant-data-of-anomaly-into-alert-message/130489/5 "2018-05-07T06:04:03Z")

</div>

Thank you for your help, I think I got it now!

I have a few additional questions:  
In result\_type: records documents, there is a "causes"-array. In that array, there are two arrays called "typical" and "actual". Is there an actual use-case why they are arrays and not just single value variables?

Same question goes to "influencers"-array, if the "influencer\_field\_name" is a single string variable, what would be the case where "influencer\_field\_values" would have multiple items?

For example, in my case it looks like this:

> "influencers": [  
> {  
> "influencer\_field\_name": "username.keyword",  
> "influencer\_field\_values": [  
> "backupuser"  
> ]  
> },  
> {  
> "influencer\_field\_name": "keywords.keyword",  
> "influencer\_field\_values": [  
> "Audit Success"  
> ]  
> }  
> ]

But what if there would be another user in addition to "backupuser" and another influencer field value in keywords.keyword, let's say "Audit Failure". How would I know which user had which anomaly?

Cheers!

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [May 7, 2018, 8:40pm UTC](https://discuss.elastic.co/t/how-to-get-all-relevant-data-of-anomaly-into-alert-message/130489/6 "2018-05-07T20:40:21Z")

</div>

In the `causes` array, the reason why `typical` and `actual` values for each "cause" is an array rather than a single value is because if you are using the `lat_long` function, there are two values for each (a latitude and a longitude).

And, of course, within the `causes` array, there may be more than one instance of a "cause", thus requiring the whole thing to be an array.

As for `influencers`, there can be more than one "influencer" and yes, there can be more than one `influencer_field_values` for a given "influencer".

To help illustrate this point, here's an anomaly record for a job that is `count by status over clientip` with influencers (`clientip, status, uri`):

```auto
      {
        "_index": ".ml-anomalies-shared",
        "_type": "doc",
        "_id": "gallery_record_1499781600000_900_0_422950106_13",
        "_score": 0,
        "_source": {
          "job_id": "gallery",
          "result_type": "record",
          "probability": 2.835781239094336e-7,
          "record_score": 50.30613,
          "initial_record_score": 88.68071046303554,
          "bucket_span": 900,
          "detector_index": 0,
          "is_interim": false,
          "timestamp": 1499781600000,
          "by_field_name": "status",
          "function": "count",
          "function_description": "count",
          "over_field_name": "clientip",
          "over_field_value": "xx.157.32.164",
          "causes": [
            {
              "probability": 0.0000028967136719120617,
              "by_field_name": "status",
              "by_field_value": "304",
              "function": "count",
              "function_description": "count",
              "typical": [
                1
              ],
              "actual": [
                272
              ],
              "over_field_name": "clientip",
              "over_field_value": "xx.157.32.164"
            },
            {
              "probability": 0.003987181891731438,
              "by_field_name": "status",
              "by_field_value": "200",
              "function": "count",
              "function_description": "count",
              "typical": [
                8.986571392274634
              ],
              "actual": [
                238
              ],
              "over_field_name": "clientip",
              "over_field_value": "xx.157.32.164"
            },
            {
              "probability": 0.015983024954163913,
              "by_field_name": "status",
              "by_field_value": "302",
              "function": "count",
              "function_description": "count",
              "typical": [
                2.117091377538835
              ],
              "actual": [
                11
              ],
              "over_field_name": "clientip",
              "over_field_value": "xx.157.32.164"
            }
          ],
          "influencers": [
            {
              "influencer_field_name": "status",
              "influencer_field_values": [
                "304"
              ]
            },
            {
              "influencer_field_name": "clientip",
              "influencer_field_values": [
                "xx.157.32.164"
              ]
            }
          ],
          "clientip": [
            "xx.157.32.164"
          ],
          "status": [
            "304"
          ]
        }
      }

```

Notice the entire record is oriented around the "over\_field\_value": "xx.157.32.164"

Hope this helps

---

<div class="post-metadata">

**Author:** ![admlko](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/admlko/32/21787_2.png) [@admlko](https://discuss.elastic.co/u/admlko)\
**Post date:** [May 8, 2018, 5:51am UTC](https://discuss.elastic.co/t/how-to-get-all-relevant-data-of-anomaly-into-alert-message/130489/7 "2018-05-08T05:51:00Z")

</div>

> if you are using the lat\_long function, there are two values for each (a latitude and a longitude).

Makes perfect sense, didn't think of that!

> Notice the entire record is oriented around the "over\_field\_value": "xx.157.32.164"

Of course, now I got it.

Thank you @richcollier for your helpful answers!

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [October 30, 2018, 10:21pm UTC](https://discuss.elastic.co/t/how-to-get-all-relevant-data-of-anomaly-into-alert-message/130489/8 "2018-10-30T22:21:34Z")

</div>