# How to find data anomalies? **URL:** \ **Category:** Elasticsearch\ **Tags:** elastic-stack-machine-learning\ **Created:** [May 19, 2022, 8:55pm UTC](https://discuss.elastic.co/t/how-to-find-data-anomalies/305238 "2022-05-19T20:55:26Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![AdamYS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adamys/32/105945_2.png) [@AdamYS](https://discuss.elastic.co/u/AdamYS)\ **Post date:** [May 19, 2022, 8:55pm UTC](https://discuss.elastic.co/t/how-to-find-data-anomalies/305238/1 "2022-05-19T20:55:26Z")
We have various indexes in Elasticsearch and I wonder if it'd be possible to find anomalies in the data? For example: We have a document for each test we run. The document has the test name as one field and the result (pass/fail) as another. How can I find the tests that frequently change their result most frequently? More complex case: We also have the commit that we ran the test on in a separate field. Can I find test name and commit combinations where the results change? Another example: We record API response times and I want to find the queries where the response time has a large variance and find what could cause that variance (e.g. an unhealthy k8s pod). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 19, 2022, 11:07pm UTC](https://discuss.elastic.co/t/how-to-find-data-anomalies/305238/2 "2022-05-19T23:07:30Z")
Welcome to our community! 😃 Take a look at [Anomaly detection | Machine Learning in the Elastic Stack [8.2] | Elastic](https://www.elastic.co/guide/en/machine-learning/current/ml-ad-overview.html) 🙂 ---
**Author:** ![AdamYS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adamys/32/105945_2.png) [@AdamYS](https://discuss.elastic.co/u/AdamYS)\ **Post date:** [May 20, 2022, 6:11am UTC](https://discuss.elastic.co/t/how-to-find-data-anomalies/305238/3 "2022-05-20T06:11:31Z")
Hi Mark! Can you give me more guidance? To be honest, machine learning seems like an overkill for the first example I gave. ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [May 20, 2022, 8:34am UTC](https://discuss.elastic.co/t/how-to-find-data-anomalies/305238/4 "2022-05-20T08:34:02Z")
It's not overkill, that's how anomaly detection is done in our stack. ---
**Author:** ![AdamYS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adamys/32/105945_2.png) [@AdamYS](https://discuss.elastic.co/u/AdamYS)\ **Post date:** [May 20, 2022, 8:40am UTC](https://discuss.elastic.co/t/how-to-find-data-anomalies/305238/5 "2022-05-20T08:40:27Z")
How could I make this work with Machine Learning in ES? > [@AdamYS](#): > > We have a document for each test we run. The document has the test name as one field and the result (pass/fail) as another. How can I find the tests that frequently change their result most frequently? ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [May 20, 2022, 1:22pm UTC](https://discuss.elastic.co/t/how-to-find-data-anomalies/305238/6 "2022-05-20T13:22:35Z")
> [@AdamYS](#): > > We have a document for each test we run. The document has the test name as one field and the result (pass/fail) as another. How can I find the tests that frequently change their result most frequently? Do you mean that the number of transitions for pass-to-fail or fail-to-pass is what is interesting here? For example, if in a particular time frame: Test A: pass, fail, pass, fail, pass, fail, pass = 6 state changes Test B: pass, pass, pass, pass, pass, pass = 0 state changes Test C: fail, fail, fail, fail, fail, fail = 0 state changes Then you're interested in highlighting Test A because it is "flip-flopping"? Pleases clarify ---
**Author:** ![AdamYS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adamys/32/105945_2.png) [@AdamYS](https://discuss.elastic.co/u/AdamYS)\ **Post date:** [May 20, 2022, 1:48pm UTC](https://discuss.elastic.co/t/how-to-find-data-anomalies/305238/7 "2022-05-20T13:48:07Z")
Exactly Rich, I'm looking to find tests that "flip-flop" the most. I'd love to be able to visualize the "top flip-flopping tests": Test Foo: 6 state changes Test Bar: 2 state change Test Baz: 1 state change ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [May 20, 2022, 2:27pm UTC](https://discuss.elastic.co/t/how-to-find-data-anomalies/305238/8 "2022-05-20T14:27:36Z")
Ok, that clarification helps. Can you also tell me how often the tests run? At a predictable interval? ---
**Author:** ![AdamYS](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/adamys/32/105945_2.png) [@AdamYS](https://discuss.elastic.co/u/AdamYS)\ **Post date:** [May 20, 2022, 2:37pm UTC](https://discuss.elastic.co/t/how-to-find-data-anomalies/305238/9 "2022-05-20T14:37:54Z")
It's not a predictable interval, but we don't have to be extremely accurate on the timing. The important part is that we would be able to look at flips for a certain commit and test combination, preferably aggregated by test name (and not the commits). ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [May 20, 2022, 3:58pm UTC](https://discuss.elastic.co/t/how-to-find-data-anomalies/305238/11 "2022-05-20T15:58:50Z")
Ok - so I think this should be done with a [Transform](https://www.elastic.co/guide/en/elasticsearch/reference/current/transforms.html), which will `pivot` on the test name and then use a `scripted_metric` aggregation to gather up all of the individual test runs, sort them by time, then count the number of times the test result "flips state" (from "pass to fail" or "fail to pass"). Here's an example based on the other examples [here](https://www.elastic.co/guide/en/elasticsearch/reference/current/transform-painless-examples.html): Note: this uses the `_preview` endpoint of the transform, but you can have the Transform persist the results to an index to use for reporting, etc. First, I define an index to use ```auto PUT testruns/ { "mappings": { "properties": { "testrundate": { "type": "date", "format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis" }, "testnamne": { "type": "keyword" }, "result": { "type": "keyword" } } } } ``` Next, I put some sample docs in there: ```auto PUT testruns/_doc/1 { "testrundate": "2022-05-20 12:00:00", "testnamne": "foo", "result": "pass" } PUT testruns/_doc/2 { "testrundate": "2022-05-20 13:00:00", "testnamne": "foo", "result": "fail" } PUT testruns/_doc/3 { "testrundate": "2022-05-20 14:00:00", "testnamne": "foo", "result": "pass" } PUT testruns/_doc/4 { "testrundate": "2022-05-20 13:00:00", "testnamne": "bar", "result": "fail" } PUT testruns/_doc/5 { "testrundate": "2022-05-20 14:00:00", "testnamne": "bar", "result": "fail" } PUT testruns/_doc/6 { "testrundate": "2022-05-20 11:00:00", "testnamne": "foo", "result": "fail" } ``` Now I run the Transform: ```auto POST _transform/_preview { "source": { "index": [ "testruns" ] }, "pivot": { "group_by": { "testnamne": { "terms": { "field": "testnamne" } } }, "aggregations": { "num_results": { "value_count": { "field": "result" } }, "flipflops" : { "scripted_metric": { "init_script": "state.docs = []", "map_script": """ Map testrun = [ 'testrundate':doc['testrundate'].value, 'testnamne':doc['testnamne'].value, 'result':doc['result'].value ]; state.docs.add(testrun) """, "combine_script": "return state.docs;", "reduce_script": """ def all_docs = []; for (s in states) { for (testrun in s) { all_docs.add(testrun); } } all_docs.sort((HashMap o1, HashMap o2)->o1['testrundate'].toEpochMilli().compareTo(o2['testrundate'].toEpochMilli())); def size = all_docs.size(); def min_time = all_docs[0]['testrundate']; def max_time = all_docs[size-1]['testrundate']; def duration = max_time.toEpochMilli() - min_time.toEpochMilli(); def last_testresult = ''; def flipflopcount = 0L; for (s in all_docs) {if (s.result != (last_testresult)) {flipflopcount++; last_testresult = s.result;}} def ret = new HashMap(); ret['first_time'] = min_time; ret['last_time'] = max_time; ret['duration_minutes'] = duration/1000/60; ret['flipflopcount'] = flipflopcount-1; return ret; """ } } } } } ``` And the output is: ```auto "preview" : [ { "testnamne" : "bar", "num_results" : 2, "flipflops" : { "duration_minutes" : 60, "first_time" : "2022-05-20T13:00:00.000Z", "last_time" : "2022-05-20T14:00:00.000Z", "flipflopcount" : 0 } }, { "testnamne" : "foo", "num_results" : 4, "flipflops" : { "duration_minutes" : 180, "first_time" : "2022-05-20T11:00:00.000Z", "last_time" : "2022-05-20T14:00:00.000Z", "flipflopcount" : 3 } } ], ``` This makes sense as test `foo` had 4 runs and transitioned 3 times and test `bar` ran twice and never transitioned (it was always failing) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 4, 2022, 8:35am UTC](https://discuss.elastic.co/t/how-to-find-data-anomalies/305238/12 "2022-11-04T08:35:36Z")