# ML: difference between partition\_field\_name and by\_field\_name in a population job?

**URL:** <https://discuss.elastic.co/t/ml-difference-between-partition-field-name-and-by-field-name-in-a-population-job/287792>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-machine-learning\
**Created:** [October 27, 2021, 12:04pm UTC](https://discuss.elastic.co/t/ml-difference-between-partition-field-name-and-by-field-name-in-a-population-job/287792 "2021-10-27T12:04:42Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![shabnam](https://avatars.discourse-cdn.com/v4/letter/s/ecae2f/32.png) [@shabnam](https://discuss.elastic.co/u/shabnam)\
**Post date:** [October 27, 2021, 12:04pm UTC](https://discuss.elastic.co/t/ml-difference-between-partition-field-name-and-by-field-name-in-a-population-job/287792/1 "2021-10-27T12:04:42Z")

</div>

Hi,  
I want to know the difference of this two ml advance jobs:  
a. **func(x),** **by\_field(y), over\_field(z),** partition\_field(-)  
b. **func(x),** by\_field(-), **over\_field(z), partition\_field(y)**

, two tests performed:

1. I created two jobs with above configuration and let them for about 2.5 month learning with func(x): 200 (constant), then I passed them events like before and events with new y and z fields values and func(x)= 200 (constant). The result shows no anomaly for new values in both 2 jobs!  
(I expected detected anomalies for first job when received new y filed value)

2. I created two other jobs with above configuration and let them for about 2.5 month learning with func(x): 200 (constant), then I passed them events like before and events with new y and z fields values and func(x)= 2000 (for this new y and z fields values). The result shows no anomaly for new values in both 2 jobs!  
(I expected at least detected anomalies for first job when received new y filed value with func(x)=2000!)

so my another question is: "When does ml population job detect a new entity as an anomaly? does it depend to type of split (soft/hard)?"

and the last question: "What is the priority of this 3 in running and scoring? by\_field, over\_field, and partition\_field?"

Thanks

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [October 27, 2021, 1:59pm UTC](https://discuss.elastic.co/t/ml-difference-between-partition-field-name-and-by-field-name-in-a-population-job/287792/2 "2021-10-27T13:59:28Z")

</div>

The role of a population job is NOT to detect new entities in a population - it is to detect entities behaving differently from their peers . If you want something like detecting new entities, then choose a temporal (non-population) job and choose `by_field` as the split (Relevant: [ML Kibana: difference between by\_field\_name and partition\_field\_name](https://discuss.elastic.co/t/ml-kibana-difference-between-by-field-name-and-partition-field-name/193385) and my comment on the notion of "dawn of time")

Also relevant: [Temporal vs. Population Analysis in Elastic Machine Learning | Elastic Blog](https://www.elastic.co/blog/temporal-vs-population-analysis-in-elastic-machine-learning)

Alternatively, if you want to detect something novel, you can consider using the `rare` function. See : [Dec 4th, 2018: [EN][ML] Rarity Analysis with Machine Learning](https://discuss.elastic.co/t/dec-4th-2018-en-ml-rarity-analysis-with-machine-learning/158979)

---

<div class="post-metadata">

**Author:** ![shabnam](https://avatars.discourse-cdn.com/v4/letter/s/ecae2f/32.png) [@shabnam](https://discuss.elastic.co/u/shabnam)\
**Post date:** [October 30, 2021, 5:16am UTC](https://discuss.elastic.co/t/ml-difference-between-partition-field-name-and-by-field-name-in-a-population-job/287792/3 "2021-10-30T05:16:47Z")

</div>

Thank you

I read the links, my question is:

**1. as "dawn of time" for new entities is meaningless in population jobs and as population job scoring is based on comparing with collective model of all peers as witnessed over time, so using "by\_field" and "over\_field" caused "by\_field" just act as a splitter and not effect on splits scoring.**

is it true?  
if it's true, so:

**2. the result of using {"by\_field(x')"+"over\_field(y')"} is the same as using {"partition\_field(x')" + "over\_field(y')"}!**

is it true?

Thanks

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [November 1, 2021, 6:39pm UTC](https://discuss.elastic.co/t/ml-difference-between-partition-field-name-and-by-field-name-in-a-population-job/287792/4 "2021-11-01T18:39:53Z")

</div>

Fact 1: Splitting a population job (which defines an `over_field`) ultimately creates sub-populations  
Fact 2: Using a `partition_field` is a more "hard" split (meant to separate/isolate) from other values - and in population analysis, you may want to isolate sub-populations from each other.  
Fact 3: Using `by_field` is more of a "soft\_split" (where values of the `by_field` are more like attributes of an entity) and anomalies of distinct members of the population are aggregated in such a way that severity of anomalousness for an entity is increased with more simultaneously unusual values for the same member of the population.

For example, imagine a data set:

```auto
time,user, gender,feature_name,feature_value
0,Bob,male,age,30
0,Bob,male,weight,175
0,Bob,male,height,75
1, Sakura,female,age,44
1, Sakura,female,weight,105
1, Sakura,female,height,59
...

```

You could set up an analysis like:  
`max(feature_value) by_field=feature_name partition_field=gender over_field=user`

where:

- `over_field=user` - makes sense since we want to model users as members of a population
- `by_field=feature_name` - age, weight, height are attributes of a particular user
- `partition_field=gender` - might make sense to isolate genders from each other because (in general) men are generally bigger/heavier than women.

---

<div class="post-metadata">

**Author:** ![shabnam](https://avatars.discourse-cdn.com/v4/letter/s/ecae2f/32.png) [@shabnam](https://discuss.elastic.co/u/shabnam)\
**Post date:** [November 2, 2021, 7:41am UTC](https://discuss.elastic.co/t/ml-difference-between-partition-field-name-and-by-field-name-in-a-population-job/287792/5 "2021-11-02T07:41:02Z")

</div>

Thanks a lot.

I would also like to know **the operation sequence order (priority) of these fields("by-field", "over\_field", and "partition\_field")** when they are all set.  
I think at first, "hard-split" is the thing the operation goes through, then "soft-split", and at the last "population" runs for splitting and anomaly scoring. Is it true?  
As you already know, the result would be different if order changed.

Thanks

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [November 2, 2021, 2:02pm UTC](https://discuss.elastic.co/t/ml-difference-between-partition-field-name-and-by-field-name-in-a-population-job/287792/6 "2021-11-02T14:02:46Z")

</div>

There's not really an "order of operations" per se, because `partition_field` and `by_field` are optional parameters. If an `over_field` is used, it is population analysis. If it is not used, it is not a population analysis.

Here's the way I think about the "formula" for a detector:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/d/8/d8eef5af98943ed664c01ae5834424277cbc54b2.png)

---

<div class="post-metadata">

**Author:** ![shabnam](https://avatars.discourse-cdn.com/v4/letter/s/ecae2f/32.png) [@shabnam](https://discuss.elastic.co/u/shabnam)\
**Post date:** [November 7, 2021, 6:04am UTC](https://discuss.elastic.co/t/ml-difference-between-partition-field-name-and-by-field-name-in-a-population-job/287792/7 "2021-11-07T06:04:34Z")

</div>

Hello

Thanks,  
In first post, 2nd test, why ml jobs didn't detect any anomaly as events with func(x)=200 normally was seen and some events with func(x)= 2000 appeared? It looks like an anomaly.

Thank you

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [November 8, 2021, 12:49pm UTC](https://discuss.elastic.co/t/ml-difference-between-partition-field-name-and-by-field-name-in-a-population-job/287792/8 "2021-11-08T12:49:27Z")

</div>

I just created a simple CSV with 1000 rows:

> **[shabnam.csv](https://drive.google.com/file/d/1SN9avwLzbFrA8prj9KiFIG5m0cAxxAZI/view?usp=sharing)**
>
> Google Drive file.

I put an anomaly in row 950:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/3/7/377dc6d5872784f7402e6cd20010ee77d84abf58.png)

I created a job with `sum(x) over z partitionfield=y`

I found the anomaly no problem:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/3/a/3a98ff2164772b907050da32e59ad93b803a0b41.png)

---

<div class="post-metadata">

**Author:** ![shabnam](https://avatars.discourse-cdn.com/v4/letter/s/ecae2f/32.png) [@shabnam](https://discuss.elastic.co/u/shabnam)\
**Post date:** [November 9, 2021, 7:29am UTC](https://discuss.elastic.co/t/ml-difference-between-partition-field-name-and-by-field-name-in-a-population-job/287792/9 "2021-11-09T07:29:35Z")

</div>

Hi  
Thank you  
I probably made a mistake somewhere.

Thanks

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 7, 2021, 7:30am UTC](https://discuss.elastic.co/t/ml-difference-between-partition-field-name-and-by-field-name-in-a-population-job/287792/10 "2021-12-07T07:30:01Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.