# ML job does not fairly distribute via all ML nodes

**URL:** <https://discuss.elastic.co/t/ml-job-does-not-fairly-distribute-via-all-ml-nodes/189638>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-machine-learning\
**Created:** [July 10, 2019, 1:03am UTC](https://discuss.elastic.co/t/ml-job-does-not-fairly-distribute-via-all-ml-nodes/189638 "2019-07-10T01:03:07Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![jasony](https://avatars.discourse-cdn.com/v4/letter/j/a3d4f5/32.png) [@jasony](https://discuss.elastic.co/u/jasony)\
**Post date:** [July 10, 2019, 1:03am UTC](https://discuss.elastic.co/t/ml-job-does-not-fairly-distribute-via-all-ml-nodes/189638/1 "2019-07-10T01:03:07Z")

</div>

hello,

i am using ML function and set 3 nodes as ML node. First i created 3 ML jobs, the cluster spread 3 jobs into all 3 nodes. but after I restarted the cluster, 3 jobs were allocated into only one node.

**Is there way to manually assign ML job into specific node?**

Please advise. Thank you!

---

<div class="post-metadata">

**Author:** ![BenTrent](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bentrent/32/33915_2.png) [@BenTrent](https://discuss.elastic.co/u/BenTrent)\
**Post date:** [July 10, 2019, 3:51pm UTC](https://discuss.elastic.co/t/ml-job-does-not-fairly-distribute-via-all-ml-nodes/189638/2 "2019-07-10T15:51:35Z")

</div>

Hey @jasony

When jobs first start, they are assigned to the ML enabled node with the least load. Specifically, load related directly to ML (memory usage, number of jobs, etc.).

When you restarted your cluster, I am guessing at a certain point all the nodes running the ML jobs stopped and only one node was available for the tasks to run. Consequently, they all got reallocated to that one node.

We don't have any formal mechanisms for re-assigning jobs. The best solution would be to stop the jobs, and start them again. They should pick up where they left off and get re-assigned to the currently least-loaded node (one of the nodes without ML jobs assigned).

---

<div class="post-metadata">

**Author:** ![jasony](https://avatars.discourse-cdn.com/v4/letter/j/a3d4f5/32.png) [@jasony](https://discuss.elastic.co/u/jasony)\
**Post date:** [July 11, 2019, 1:29am UTC](https://discuss.elastic.co/t/ml-job-does-not-fairly-distribute-via-all-ml-nodes/189638/3 "2019-07-11T01:29:27Z")

</div>

thank you for your advice. based on your advise, i stopped and started jobs then saw all jobs were re-assigned fairly.

thank you!

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 8, 2019, 1:29am UTC](https://discuss.elastic.co/t/ml-job-does-not-fairly-distribute-via-all-ml-nodes/189638/4 "2019-08-08T01:29:28Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.