# Less number of active ML Node in anomaly detection jobs

**URL:** <https://discuss.elastic.co/t/less-number-of-active-ml-node-in-anomaly-detection-jobs/254827>\
**Category:** Elasticsearch\
**Tags:** elastic-stack-machine-learning\
**Created:** [November 9, 2020, 11:07pm UTC](https://discuss.elastic.co/t/less-number-of-active-ml-node-in-anomaly-detection-jobs/254827 "2020-11-09T23:07:48Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![heric](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/heric/32/70748_2.png) [@heric](https://discuss.elastic.co/u/heric)\
**Post date:** [November 9, 2020, 11:07pm UTC](https://discuss.elastic.co/t/less-number-of-active-ml-node-in-anomaly-detection-jobs/254827/1 "2020-11-09T23:07:48Z")

</div>

Hi,

I have 5 nodes elasticsearch cluster and for all of them i activated ML roles.  
from GET \_nodes i can see ml role for each of the nodes, however in the anomaly detection jobs i saw only 4 active ML nodes

```auto

      "roles" : [
        "data",
        "ingest",
        "ml",
        "remote_cluster_client",
        "transform"
      ],

```

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/f/6f084479aefdcfe14b5703a4d9b390a9161fb3b1.png)

what can i check to see what is wrong ?

---

<div class="post-metadata">

**Author:** ![darnautov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/darnautov/32/70608_2.png) [@darnautov](https://discuss.elastic.co/u/darnautov)\
**Post date:** [November 10, 2020, 9:31am UTC](https://discuss.elastic.co/t/less-number-of-active-ml-node-in-anomaly-detection-jobs/254827/2 "2020-11-10T09:31:15Z")

</div>

Hi @heric,

Just to clarify in case the naming is confusing, the "Active ML nodes" counter does not reflect the number of nodes with the "ml" role assigned. It is based on the running jobs on certain nodes, hence called "Active". So it means there are no active jobs allocated on one of your ml nodes.

---

<div class="post-metadata">

**Author:** ![heric](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/heric/32/70748_2.png) [@heric](https://discuss.elastic.co/u/heric)\
**Post date:** [November 10, 2020, 1:15pm UTC](https://discuss.elastic.co/t/less-number-of-active-ml-node-in-anomaly-detection-jobs/254827/3 "2020-11-10T13:15:49Z")

</div>

Hi @darnautov , Thank you for the clarification.

I will add more jobs to see if the active ML nodes increase.

Is there any way i can check which jobs is assigned to which nodes ?

---

<div class="post-metadata">

**Author:** ![darnautov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/darnautov/32/70608_2.png) [@darnautov](https://discuss.elastic.co/u/darnautov)\
**Post date:** [November 10, 2020, 3:02pm UTC](https://discuss.elastic.co/t/less-number-of-active-ml-node-in-anomaly-detection-jobs/254827/4 "2020-11-10T15:02:07Z")

</div>

@heric, yes, using UI you can expand a row on the "Anomaly detection jobs" page and see the node name

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/a/8aecf4d853490874687209da14050df25b62b81a.png)

You can also refer to [the anomaly detection job statistics API](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-get-job-stats.html).

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [November 10, 2020, 3:03pm UTC](https://discuss.elastic.co/t/less-number-of-active-ml-node-in-anomaly-detection-jobs/254827/5 "2020-11-10T15:03:28Z")

</div>

Example of using the API

contents of `list_all_ml_jobs_nodes.sh`:

```auto
#!/bin/bash
HOST='localhost'
PORT=9200
#CURL_AUTH="-u elastic:changeme"
#get list of all jobs
list=`curl $CURL_AUTH -s http://$HOST:$PORT/_ml/anomaly_detectors?pretty | awk -F" : " '/job_id/{print $2}' | sed 's/\",//g' | sed 's/\"//g'`
echo "job_id,node"
#loop through all jobs to find which node it is running on
while read -r JOB_ID; do
   curl $CURL_AUTH -s -XGET $HOST:$PORT/_ml/anomaly_detectors/${JOB_ID}/_stats?pretty | jq '{job_id: .jobs[0].job_id, node: .jobs[0].node.name}' | jq --raw-output '"\(.job_id),\(.node)"'
done <<< "$list"

```

Would yield an output something like:

```auto
%> ./list_all_ml_jobs_nodes.sh
job_id,node
aaaaaaa,richs-mbp.lan
authd,null
bbbbb,richs-mbp.lan
bot_detection,null
...

```

where the first column is the `job_id` and the second is the node name (`null` if not running)

---

<div class="post-metadata">

**Author:** ![heric](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/heric/32/70748_2.png) [@heric](https://discuss.elastic.co/u/heric)\
**Post date:** [November 10, 2020, 4:15pm UTC](https://discuss.elastic.co/t/less-number-of-active-ml-node-in-anomaly-detection-jobs/254827/6 "2020-11-10T16:15:31Z")

</div>

After adding more Anomaly Job

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/8/980ef6d1e692b1a7dc494a0242adfd52f0d5ab3b.png)

Thank you @darnautov @richcollier

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 8, 2020, 4:15pm UTC](https://discuss.elastic.co/t/less-number-of-active-ml-node-in-anomaly-detection-jobs/254827/7 "2020-12-08T16:15:35Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.