# Anomaly detection - partitioning data **URL:** \ **Category:** Kibana\ **Tags:** elastic-stack-machine-learning\ **Created:** [October 5, 2020, 9:18am UTC](https://discuss.elastic.co/t/anomaly-detection-partitioning-data/250957 "2020-10-05T09:18:45Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![mduzy](https://avatars.discourse-cdn.com/v4/letter/m/e5b9ba/32.png) [@mduzy](https://discuss.elastic.co/u/mduzy)\ **Post date:** [October 5, 2020, 9:18am UTC](https://discuss.elastic.co/t/anomaly-detection-partitioning-data/250957/1 "2020-10-05T09:18:45Z")
Hi, I've been working with the anomaly detection functionality in Kibana. I've got about 500 records processed, all of them have a transaction.name and a custom id that works as an tenant indicator (I'll call it tenantId from now). So I create a anomaly detection job, with these detectors: `count partitionfield="transaction.name"` `count partitionfield="tenntId"` and I get it properly partitioned by all the values in these fields throughout the dataset. However when I try to use a by field, it becomes weird: `count by "tenantId" partitionfield="transaction.name"` `count by "transaction.name" partitionfield="tenantId"` I get the choice to select a combination of tenantId - transaction.name, however the dropdowns (on single metric view) are somewhat lacking in data, I can choose like 1 tenantId and 1 transaction.name. Tried to use this a few times and can't really get around this... What I'm trying to achieve here is to detect anomalies in each tenant and by API method calls because some tenants may be much more busy than others and some method calls may be more used than others. So any ideas why I can't get a full set of possible tenantIds and transaction.names in those dropdowns? ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [October 5, 2020, 11:31am UTC](https://discuss.elastic.co/t/anomaly-detection-partitioning-data/250957/2 "2020-10-05T11:31:47Z")
Please report the version you are using when you post questions. This is relevant specifically in this case because the behavior of the UI has changed with respect to this over time (see [https://github.com/elastic/kibana/issues/52618](https://github.com/elastic/kibana/issues/52618) for example). Also, adding screenshots to your posts is very helpful for us. ---
**Author:** ![mduzy](https://avatars.discourse-cdn.com/v4/letter/m/e5b9ba/32.png) [@mduzy](https://discuss.elastic.co/u/mduzy)\ **Post date:** [October 5, 2020, 12:50pm UTC](https://discuss.elastic.co/t/anomaly-detection-partitioning-data/250957/3 "2020-10-05T12:50:22Z")
I'm working with version 7.9.2 ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [October 5, 2020, 3:04pm UTC](https://discuss.elastic.co/t/anomaly-detection-partitioning-data/250957/4 "2020-10-05T15:04:14Z")
Thanks for the info. Now, when you say "somewhat lacking data" - I need to know what you mean by that. Because it is possible that each combination of `transaction.name` and `tenantId ` results in sparse data just by its nature. Are you sure you really need the double-split here? Why not just the single split (perhaps partitioning on `transaction.name` and just leaving `tenantId ` as an influencer? ---
**Author:** ![mduzy](https://avatars.discourse-cdn.com/v4/letter/m/e5b9ba/32.png) [@mduzy](https://discuss.elastic.co/u/mduzy)\ **Post date:** [October 6, 2020, 8:08am UTC](https://discuss.elastic.co/t/anomaly-detection-partitioning-data/250957/5 "2020-10-06T08:08:03Z")
So when data is sparse, the options may not show in the dropdowns? So maybe if all the tenants generate enough data it will show everything? Because it behaves awkwardly - when I do `count by "tenantId" partitionfield="transaction.name"` it returns only tenantId=2 and one transaction.name, and when I reverse it - `count by "transaction.name" partitionfield="tenantId"` I get tenantId=52 and a different transaction.name. Unfortunately it's a requirement for the project I'm working on to have the double split here. ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [October 6, 2020, 11:54am UTC](https://discuss.elastic.co/t/anomaly-detection-partitioning-data/250957/6 "2020-10-06T11:54:49Z")
I was cautioning about sparse data with respect to the modeling. If you "oversplit" the data, you may have a situation in which the unique combination of the `by_field` and the `partition_field` doesn't really occur very frequently, thus giving the possibility of a reduced number of observations and inadequate modeling. In the UI, the dropdowns will only show entities that have anomaly records. For example, I just did a contrived job of `count by request.keyword partitionfield=geo.src` on the sample kibana web logs data set (there are 175+ unique `request.keyword` values and 160+ unique `geo.src` values). However, on execution of my job, only 7 anomalies are found in the data set, but all for the combination of `geo.src:"CN"` and `request.keyword:"/beats/metricbeat"`. Therefore, the UI looks like: ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/c/fcf291a7832ecc719219b4aa3b6591d9f7d28edf.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/f/bf553a6ba344c63cb816dd88ce798220dea73424.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/3/434bf0ca6b104a4135224f3e51c1413a9523bce9.png) ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/5/d520f88d944329d418f4d28ba9bad8b4d441d144.png) In other words, the dropdown doesn't show any of the other values of `geo-src` or `request.keyword` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 3, 2020, 11:54am UTC](https://discuss.elastic.co/t/anomaly-detection-partitioning-data/250957/7 "2020-11-03T11:54:59Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.