# How elastic ML detect anomaly? **URL:** \ **Category:** Elasticsearch\ **Tags:** elastic-stack-machine-learning\ **Created:** [December 19, 2022, 11:09am UTC](https://discuss.elastic.co/t/how-elastic-ml-detect-anomaly/321551 "2022-12-19T11:09:26Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![Aniket\_Pant](https://avatars.discourse-cdn.com/v4/letter/a/77aa72/32.png) [@Aniket\_Pant](https://discuss.elastic.co/u/Aniket_Pant)\ **Post date:** [December 19, 2022, 11:09am UTC](https://discuss.elastic.co/t/how-elastic-ml-detect-anomaly/321551/1 "2022-12-19T11:09:27Z")
Hi Community, ![1st Screenshot](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1a4146b4262bd2c65b76def3fe107574f5c57447.png) In above screenshot the upper bound values is between 1 to 4.8 something ![2nd Screenshot](https://us1.discourse-cdn.com/elastic/original/3X/9/2/926275687c997642363618f8102fd0667fa008a1.png) On dec 12th at 9:15 it detect major anomaly it actual value was 3 , upper bound value was 0.754 and lower bound was 0 so in past data we have these values , why it consider as major anomaly not minor anomaly ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [December 21, 2022, 6:32pm UTC](https://discuss.elastic.co/t/how-elastic-ml-detect-anomaly/321551/2 "2022-12-21T18:32:44Z")
Hard to tell what is fully going on here from just a few screenshots but it seems from the first screenshot the point which is marked as point A is inside the expected bounds (the blue shadow) whereas point B is not. ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/4/5483d9affbe788d895df9b7adcd81f3335635345.png) Now, what obviously isn't possible to know from the screenshots is why the expected bounds has this hump shape at this time. Can you explain that the use case is (i.e. what the data represents) and what your detection configuration is for the ML job (i.e. what function is used, etc.) ---
**Author:** ![Aniket\_Pant](https://avatars.discourse-cdn.com/v4/letter/a/77aa72/32.png) [@Aniket\_Pant](https://discuss.elastic.co/u/Aniket_Pant)\ **Post date:** [December 26, 2022, 7:26am UTC](https://discuss.elastic.co/t/how-elastic-ml-detect-anomaly/321551/3 "2022-12-26T07:26:03Z")
Hi @richcollier Here is the Job config ```auto "analysis_config": { "bucket_span": "15m", "detectors": [ { "detector_description": "rare by \"resp.errCode.keyword\"", "function": "rare", "by_field_name": "resp.errCode.keyword", "detector_index": 0 } ], "influencers": [ "payer.name.keyword" ] }, ``` There is a field called **error code** and i am using rare function. The purpose of applying this function is that we want to detect which error code happened very rarely. ---
**Author:** ![Aniket\_Pant](https://avatars.discourse-cdn.com/v4/letter/a/77aa72/32.png) [@Aniket\_Pant](https://discuss.elastic.co/u/Aniket_Pant)\ **Post date:** [December 28, 2022, 5:40am UTC](https://discuss.elastic.co/t/how-elastic-ml-detect-anomaly/321551/4 "2022-12-28T05:40:12Z")
Hi @richcollier can you please help me to understand . ---
**Author:** ![Aniket\_Pant](https://avatars.discourse-cdn.com/v4/letter/a/77aa72/32.png) [@Aniket\_Pant](https://discuss.elastic.co/u/Aniket_Pant)\ **Post date:** [December 30, 2022, 6:46am UTC](https://discuss.elastic.co/t/how-elastic-ml-detect-anomaly/321551/5 "2022-12-30T06:46:53Z")
Can anyone tell me what this ml graph depicting. ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [January 2, 2023, 1:29am UTC](https://discuss.elastic.co/t/how-elastic-ml-detect-anomaly/321551/6 "2023-01-02T01:29:25Z")
I'm a little dubious about whether the previously shown graph is for the `rare by resp.errCode.keyword` job you provided the configuration for. This is because jobs using the rare function don't have model bounds shown with expected values as rare values don't exist routinely enough to plot a predicted value. See: ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/7/b71ff5e83fbcfbe38d04793239e00ce972903160.jpeg) Can you provide another screenshot showing the job name and detector, as mine shows? I truly suspect the previous screenshot was from a job that was modeling `count` , not `rare`. ---
**Author:** ![Aniket\_Pant](https://avatars.discourse-cdn.com/v4/letter/a/77aa72/32.png) [@Aniket\_Pant](https://discuss.elastic.co/u/Aniket_Pant)\ **Post date:** [January 2, 2023, 7:29am UTC](https://discuss.elastic.co/t/how-elastic-ml-detect-anomaly/321551/7 "2023-01-02T07:29:19Z")
Hi @richcollier , Sorry i just posted wrong ML snap This is the actual ML Job config ```auto "analysis_config": { "bucket_span": "15m", "detectors": [ { "detector_description": "max(test) by \"verb.keyword\"", "function": "max", "field_name": "test", "by_field_name": "verb.keyword", "detector_index": 0 } ], "influencers": [ "verb.keyword" ] }, ``` I am detecting max value for a particular service( **verb.keyword** ) I have recreated the ML job because earlier it was deleted mistakenly by me **Previous Pattern** ![latest](https://us1.discourse-cdn.com/elastic/original/3X/0/9/0953aeb8aff84610a2e1465f9f5e25ae6e5819f6.png) **Today's Pattern** ![Latest2](https://us1.discourse-cdn.com/elastic/original/3X/5/6/566b5dfabaa3fe43d2494d3d4619c79f8a492bee.png) One thing i have noticed that for **Previous Pattern** snap is different which i posted 2 weeks back. Does pattern revised if we delete the job ? > I have recreated the ML job because earlier it was deleted mistakenly by me **Previous Pattern** ![latest](https://us1.discourse-cdn.com/elastic/original/3X/0/9/0953aeb8aff84610a2e1465f9f5e25ae6e5819f6.png) ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [January 2, 2023, 5:41pm UTC](https://discuss.elastic.co/t/how-elastic-ml-detect-anomaly/321551/8 "2023-01-02T17:41:18Z")
> [@Aniket\_Pant](#): > > I am detecting max value for a particular service( **verb.keyword** ) > > I have recreated the ML job because earlier it was deleted mistakenly by me Ok, that makes more sense. Thanks for clarifying. > One thing i have noticed that for **Previous Pattern** snap is different which i posted 2 weeks back. Does pattern revised if we delete the job ? If you run the same data (and amount of data) through the same job configuration, you will get the same results. If you send less data, more data, etc. you will get different results. In your screenshots above, I'm now lost as to what you're trying to ask. In the one labeled "Previous Pattern" shows results from early in the analysis/learning cycle (remember data is processed in chronological order so data from 12-10-2022 is "early") - so therefore, the predictions and anomalies will be less accurate/meaningful than those seen later in the data (around 12-30-2022, for example). This is why the blue shading more accurately follows the expected ranges of the values of the data in late December than early December - as ML has had more data to discover the natural pattern of the data. Is this what you were wondering? ---
**Author:** ![Aniket\_Pant](https://avatars.discourse-cdn.com/v4/letter/a/77aa72/32.png) [@Aniket\_Pant](https://discuss.elastic.co/u/Aniket_Pant)\ **Post date:** [January 3, 2023, 5:44am UTC](https://discuss.elastic.co/t/how-elastic-ml-detect-anomaly/321551/9 "2023-01-03T05:44:56Z")
I have two question what **+** indicate in many blogs and documentation it say it is multi-bucket can you tell me what actually it is by referring my snap and second there is annotation also but there is no reason for annotation ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [January 3, 2023, 4:07pm UTC](https://discuss.elastic.co/t/how-elastic-ml-detect-anomaly/321551/10 "2023-01-03T16:07:35Z")
If you read the blog [Interpreting multi-bucket impact anomalies using Elastic machine learning features | Elastic Blog](https://www.elastic.co/blog/interpreting-multi-bucket-impact-anomalies-using-elastic-machine-learning-features) you will see that the `+` icon indeed indicates that the anomaly has a significant multi-bucket factor to it - meaning that it's not just the individual point that may be anomalous, but there are a series of consecutive points that are anomalous together (derived from a sliding window of 12 bucket spans). As for your other comment about the annotation - what does the annotation say? ---
**Author:** ![Aniket\_Pant](https://avatars.discourse-cdn.com/v4/letter/a/77aa72/32.png) [@Aniket\_Pant](https://discuss.elastic.co/u/Aniket_Pant)\ **Post date:** [January 4, 2023, 10:23am UTC](https://discuss.elastic.co/t/how-elastic-ml-detect-anomaly/321551/11 "2023-01-04T10:23:22Z")
@richcollier Thank you for really sharing your knowledge but in machine learning i m just stuck at shaded part that is trend or model.Sometime shaded blue will shrink or wider does it tell something or is there any prior concepts should i know to depit this waves in graph. ![image](https://us1.discourse-cdn.com/elastic/original/3X/0/b/0b677a18ae1cfc756556ceb96afdf2dcf5393b4d.png) > As for your other comment about the annotation - what does the annotation say? > it just say "Change in trend" Yeahh but really thank you. ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [January 4, 2023, 5:01pm UTC](https://discuss.elastic.co/t/how-elastic-ml-detect-anomaly/321551/12 "2023-01-04T17:01:14Z")
Data is processed in a streaming manner in chronological order. So, the data on the left hand side is modeled very coarsely until the time where you see annotations 2,3, and 4 on the picture. This is when the model recognizes the up-and-down "sawtooth" trend and the range of expected values (as depicted by the blue shading) more accurately follows the data. This is a very typical thing that happens if and when the data you're modeling with ML detects a periodic trend. See another example: ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/6/a66699068976c8dae2a5290e7f7729317a541011.png) You should know that you shouldn't really "trust" the results/anomalies that are flagged early in the analysis (way on the left-hand side). You should ensure that a least a few days (and maybe as many as 3 weeks) of data is processed by the ML job so that all typical periodic trends (hourly, daily, weekly) can be properly identified. If you want to learn more about ML, pick up a copy of my book: - **For sale on packt:** [Search | Packt Subscription](https://www.packtpub.com/big-data-and-business-intelligence/machine-learning-elastic-stack) - **For sale on Amazon:** [https://www.amazon.com/Rich-Collier/e/B07NF94RDQ](https://www.amazon.com/Rich-Collier/e/B07NF94RDQ?ref_=dbs_p_pbk_r00_abau_000000) - **Register for free e-Book:** [Digital book download: Machine Learning with the Elastic Stack - Second Edition](https://events.elastic.co/machinelearningwithelastic) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 1, 2023, 5:01pm UTC](https://discuss.elastic.co/t/how-elastic-ml-detect-anomaly/321551/13 "2023-02-01T17:01:52Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.