# \[Machine Learning\] No document detection **URL:** \ **Category:** Elasticsearch\ **Tags:** elastic-stack-machine-learning\ **Created:** [June 26, 2019, 8:25am UTC](https://discuss.elastic.co/t/machine-learning-no-document-detection/187519 "2019-06-26T08:25:33Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![fbonningues](https://avatars.discourse-cdn.com/v4/letter/f/e9a140/32.png) [@fbonningues](https://discuss.elastic.co/u/fbonningues)\ **Post date:** [June 26, 2019, 8:25am UTC](https://discuss.elastic.co/t/machine-learning-no-document-detection/187519/1 "2019-06-26T08:25:33Z")
Hi, I have activated the trial licence to make some test with Machine Learning feature. I have a continus data feed with defined templates, each document having a field with a specific value (in this case, a shop\_id). The data feed pattern is to receive data throughout the day, on a typical basis : 7AM-9PM. The current depth of data is more than a month, for almost 250M documents. I want to detect alimentation failure for a specific time series (with shop\_id as query\_key), so a detection of 'no document received'. I tried to create a job with 'Population' wizard and used it for data in which I knew some documents are missing, but that was not seen as an anomaly. In the documentation, I didn't find any reference for that kind of detection, can you help me solve this problem ? Thx a lot François ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [June 26, 2019, 12:31pm UTC](https://discuss.elastic.co/t/machine-learning-no-document-detection/187519/2 "2019-06-26T12:31:09Z")
Yes, this type of detection is possible and would employ the `count` function (or more likely, the `low_count` function) So, do you want to detect low document counts for EVERY `shop_id` (in other words, splitting the analysis for every instance of that field) or are you asking how to filter the data so that only a certain value of `shop_id` gets considered? ---
**Author:** ![fbonningues](https://avatars.discourse-cdn.com/v4/letter/f/e9a140/32.png) [@fbonningues](https://discuss.elastic.co/u/fbonningues)\ **Post date:** [June 26, 2019, 12:46pm UTC](https://discuss.elastic.co/t/machine-learning-no-document-detection/187519/3 "2019-06-26T12:46:58Z")
Thx for the answer. In the current job, I use this detector : low\_count over shop\_id (so the function you mentionned) The time series are independent over shop\_id (independent data feed) and I want to know if one of the shops doesn't send documents. That in order to detect issues of connection, treatments, etc for each shop. In my current data, one of the shops didn't send documents on the range of time usually populated, so I expected to see an anomaly, that I not had. ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [June 26, 2019, 12:55pm UTC](https://discuss.elastic.co/t/machine-learning-no-document-detection/187519/4 "2019-06-26T12:55:05Z")
I'd suggest not using a population job and using a multi-metric job split (or using the `partition_field_name` in the API). In a population analysis - you are comparing `shop_id`s against each other (which maybe ok) but I would think that it would be better to compare a particular `shop_id` against itself over time (which a population analysis will not do). Also, if the condition is a set rule (i.e. any occurrence of zero documents) - you pretty much don't need ML - you can accomplish this with a standard Watch ---
**Author:** ![fbonningues](https://avatars.discourse-cdn.com/v4/letter/f/e9a140/32.png) [@fbonningues](https://discuss.elastic.co/u/fbonningues)\ **Post date:** [June 26, 2019, 1:03pm UTC](https://discuss.elastic.co/t/machine-learning-no-document-detection/187519/5 "2019-06-26T13:03:39Z")
"Compare a particular `shop_id` against itself over time", that exactly what I need, because each shop have a particular pattern of sells (i.e. documents sends). A standard watch wouldn't be suffisant because of the downtime of documents reception (during the night or on week end) I'll try that kind of job, and I'll keep you posted ! ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [June 26, 2019, 1:17pm UTC](https://discuss.elastic.co/t/machine-learning-no-document-detection/187519/6 "2019-06-26T13:17:47Z")
Cool By the way - here's a blog that compares the different approaches of Population analysis versus Temporal analysis: [https://www.elastic.co/blog/temporal-vs-population-analysis-in-elastic-machine-learning](https://www.elastic.co/blog/temporal-vs-population-analysis-in-elastic-machine-learning) ---
**Author:** ![fbonningues](https://avatars.discourse-cdn.com/v4/letter/f/e9a140/32.png) [@fbonningues](https://discuss.elastic.co/u/fbonningues)\ **Post date:** [June 26, 2019, 2:30pm UTC](https://discuss.elastic.co/t/machine-learning-no-document-detection/187519/7 "2019-06-26T14:30:14Z")
Thx @richcollier ! Using a multi metric job with detector : low\_count partition\_field\_name=`shop_id` worked fine ! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 24, 2019, 2:30pm UTC](https://discuss.elastic.co/t/machine-learning-no-document-detection/187519/8 "2019-07-24T14:30:14Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.