# Can we use "sub-partitioning" in ML? **URL:** \ **Category:** Elasticsearch\ **Tags:** elastic-stack-machine-learning\ **Created:** [August 31, 2017, 12:54pm UTC](https://discuss.elastic.co/t/can-we-use-sub-partitioning-in-ml/99002 "2017-08-31T12:54:32Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Vitaly\_il](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vitaly_il/32/130964_2.png) [@Vitaly\_il](https://discuss.elastic.co/u/Vitaly_il)\ **Post date:** [August 31, 2017, 12:54pm UTC](https://discuss.elastic.co/t/can-we-use-sub-partitioning-in-ml/99002/1 "2017-08-31T12:54:32Z")
Our goal is similar to discussed in this thread [Machine learning - host stopped sending logs or events](https://discuss.elastic.co/t/machine-learning-host-stopped-sending-logs-or-events/96938/8) - we're trying to send alerts when certain hosts or components stopped to send data or we have anomaly in data volume. There are about 5 indices, with about 30 log types, tens of hosts. We created multi-count job, using "type" as partition\_field\_name. But what about per-host issues? Should we use "host" field as influencer or we can use "host" as a "sub-partion"? Thank you, Vitaly ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [August 31, 2017, 1:58pm UTC](https://discuss.elastic.co/t/can-we-use-sub-partitioning-in-ml/99002/2 "2017-08-31T13:58:10Z")
Vitaly, You could consider making an Advanced Job - with a config like: function: low\_count by\_field\_name: type partition\_field\_name: host This will effectively give you a double-split - "for every host, look at the low count of data per type" ---
**Author:** ![colby](https://avatars.discourse-cdn.com/v4/letter/c/d07c76/32.png) [@colby](https://discuss.elastic.co/u/colby)\ **Post date:** [August 31, 2017, 8:20pm UTC](https://discuss.elastic.co/t/can-we-use-sub-partitioning-in-ml/99002/3 "2017-08-31T20:20:30Z")
If you add the host name to a "tags" field then you can do an advanced job and just set the datafeed to query for that tag so that each job will essentially be a partition. ---
**Author:** ![Vitaly\_il](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vitaly_il/32/130964_2.png) [@Vitaly\_il](https://discuss.elastic.co/u/Vitaly_il)\ **Post date:** [September 7, 2017, 3:01pm UTC](https://discuss.elastic.co/t/can-we-use-sub-partitioning-in-ml/99002/4 "2017-09-07T15:01:41Z")
Rich, I'll totally understand if your answer will be "RTFM", but I'll ask anyway 1. what is the difference between using MultiMetric job wizard with partition\_field\_name="type.keyword" and using "host" field as influencer vs going to Advanced job and using by\_field\_name as you suggested. I.e. will second option provide better granularity and/or will allow to create more usable alerts? 2. Does "low count" have some advantages vs just "count" for detect anomalies in number of events? I.e. is ML job with "count" can replace two jobs with "low count" and "high count"? Thank you again, Vitaly ---
**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\ **Post date:** [September 7, 2017, 4:44pm UTC](https://discuss.elastic.co/t/can-we-use-sub-partitioning-in-ml/99002/5 "2017-09-07T16:44:30Z")
1. The multi-metric job cannot do a by\_field\_name and a partition\_field\_name. It only does partition\_field\_name. A choice of an influencer does not affect the statistical modeling (i.e. won't split the modeling) - possible influencers are determined after the anomaly is determined. 2. A `count` job is equivalent to `low_count` and `high_count` within the same job. You'd pick `low_count` if you were ONLY interested in a drop in events. ---
**Author:** ![Vitaly\_il](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vitaly_il/32/130964_2.png) [@Vitaly\_il](https://discuss.elastic.co/u/Vitaly_il)\ **Post date:** [September 7, 2017, 6:20pm UTC](https://discuss.elastic.co/t/can-we-use-sub-partitioning-in-ml/99002/6 "2017-09-07T18:20:12Z")
thank you \* 2! Vitaly ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 5, 2017, 6:20pm UTC](https://discuss.elastic.co/t/can-we-use-sub-partitioning-in-ml/99002/7 "2017-10-05T18:20:18Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.