# No ML nodes with sufficient capacity for trained model deployment

**URL:** https://discuss.elastic.co/t/no-ml-nodes-with-sufficient-capacity-for-trained-model-deployment/357517
**Category:** Elasticsearch
**Tags:** elastic-stack-machine-learning
**Created:** [April 16, 2024, 12:46pm UTC](https://discuss.elastic.co/t/no-ml-nodes-with-sufficient-capacity-for-trained-model-deployment/357517 "2024-04-16T12:46:14Z")
**Posts on this page:** 10
**Page:** 1

<div class="post-metadata">

### Author: ![msola](https://avatars.discourse-cdn.com/v4/letter/m/7feea3/32.png) [@msola](https://discuss.elastic.co/u/msola)
#### Post date: [April 16, 2024, 12:46pm UTC](https://discuss.elastic.co/t/no-ml-nodes-with-sufficient-capacity-for-trained-model-deployment/357517/1 "2024-04-16T12:46:14Z")

</div>

Hi,  
I'm new with elasticsearch. I currently have the trial period Platinum subscription, since I want to check if elasticsearch fits with what I want to do.

I want to test the elster\_model\_2 ML model. However, every time that I try to deploy it I get the following error:

**Could not start deployment because no ML nodes with sufficient capacity were found**

```auto
{
  "statusCode": 429,
  "error": "Too Many Requests",
  "message": "[status_exception\n\tCaused by:\n\t\tillegal_state_exception: Could not start deployment because no suitable nodes were found, allocation explanation [Could not assign (more) allocations on node [iYF2E0RcS8S5D_rUdpSBXA]. Reason: This node has insufficient available memory. Available memory for ML [322122547 (307.1mb)], free memory [322122547 (307.1mb)], estimated memory required for this model [469581194 (447.8mb)].|Could not assign (more) allocations on node [rmiiO1-LTtaZVPon05N39Q]. Reason: This node has insufficient available memory. Available memory for ML [322122547 (307.1mb)], free memory [322122547 (307.1mb)], estimated memory required for this model [469581194 (447.8mb)].|Could not assign (more) allocations on node [z9GMuSTwRuGuDGtwQI8BGQ]. Reason: This node has insufficient available memory. Available memory for ML [322122547 (307.1mb)], free memory [322122547 (307.1mb)], estimated memory required for this model [469581194 (447.8mb)].]\n\tRoot causes:\n\t\tstatus_exception: Could not start deployment because no ML nodes with sufficient capacity were found]: Could not start deployment because no ML nodes with sufficient capacity were found",
  "attributes": {
    "body": {
      "error": {
        "root_cause": [
          {
            "type": "status_exception",
            "reason": "Could not start deployment because no ML nodes with sufficient capacity were found"
          }
        ],
        "type": "status_exception",
        "reason": "Could not start deployment because no ML nodes with sufficient capacity were found",
        "caused_by": {
          "type": "illegal_state_exception",
          "reason": "Could not start deployment because no suitable nodes were found, allocation explanation [Could not assign (more) allocations on node [iYF2E0RcS8S5D_rUdpSBXA]. Reason: This node has insufficient available memory. Available memory for ML [322122547 (307.1mb)], free memory [322122547 (307.1mb)], estimated memory required for this model [469581194 (447.8mb)].|Could not assign (more) allocations on node [rmiiO1-LTtaZVPon05N39Q]. Reason: This node has insufficient available memory. Available memory for ML [322122547 (307.1mb)], free memory [322122547 (307.1mb)], estimated memory required for this model [469581194 (447.8mb)].|Could not assign (more) allocations on node [z9GMuSTwRuGuDGtwQI8BGQ]. Reason: This node has insufficient available memory. Available memory for ML [322122547 (307.1mb)], free memory [322122547 (307.1mb)], estimated memory required for this model [469581194 (447.8mb)].]"
        }
      },
      "status": 429
    }
  }
}

```

I curently have 3 ML nodes with 1GB total memory each (512 MB estimated available memory). Am I missing something? How can I change this? Would having more RAM solve this issue (I currently only have 32 GB)?

Thank you very much!

---

<div class="post-metadata">

### Author: ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [April 16, 2024, 2:25pm UTC](https://discuss.elastic.co/t/no-ml-nodes-with-sufficient-capacity-for-trained-model-deployment/357517/2 "2024-04-16T14:25:35Z")

</div>

Hi @msola Welcome to the community!.

Well, in short, you don't have enough RAM on any one of the ml nodes to support the model.

In general, it's better to scale ml nodes vertically not horizontally so that you have more RAM in any one node.

If you are RAM constrained, it's probably a better idea to make one larger node than three smaller nodes.

1 GB nodes are very small in the ml world as each model it's going to take several hundred Mbs.

---

<div class="post-metadata">

### Author: ![msola](https://avatars.discourse-cdn.com/v4/letter/m/7feea3/32.png) [@msola](https://discuss.elastic.co/u/msola)
#### Post date: [April 16, 2024, 4:13pm UTC](https://discuss.elastic.co/t/no-ml-nodes-with-sufficient-capacity-for-trained-model-deployment/357517/3 "2024-04-16T16:13:16Z")

</div>

Thanks! The nodes were automatically set to 1GB, even if I only choose one node. How can I increase the size of one single node?

Does it mean that if I use a computer with a large RAM, the nodes will automatically have a larger memory?

---

<div class="post-metadata">

### Author: ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)
#### Post date: [April 16, 2024, 4:58pm UTC](https://discuss.elastic.co/t/no-ml-nodes-with-sufficient-capacity-for-trained-model-deployment/357517/4 "2024-04-16T16:58:16Z")

</div>

@msola How did you deploy your cluster?

On Prem Self Managed Nodes, Dockers? K8s? Are they in containers?

> [@msola](#):
>
> Does it mean that if I use a computer with a large RAM, the nodes will automatically have a larger memory?

Yes Elastic will set the RAM to 50% of the available system memory assuming that we have not changed configurations.  
Did you set up specific ML Nodes?  
You will need to share your configs otherwise we will just be guessing and answering a lot of partial questions 🙂

---

<div class="post-metadata">

### Author: ![slytherin\_wyne](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/slytherin_wyne/32/137679_2.png) [@slytherin\_wyne](https://discuss.elastic.co/u/slytherin_wyne)
#### Post date: [September 18, 2024, 7:37am UTC](https://discuss.elastic.co/t/no-ml-nodes-with-sufficient-capacity-for-trained-model-deployment/357517/5 "2024-09-18T07:37:30Z")

</div>

Hello,I got the same situation AND my cluster deployed in docker containers with 4GB RAM constrained. I got the error code when I try to test the built-in model E5.  
Should I have to redeploy a cluster with 16GB at least?  
OR I found some documents mention # autoscaling. Does it can solve this problem? How do i enable it ?

---

<div class="post-metadata">

### Author: ![dkyle](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dkyle/32/59114_2.png) [@dkyle](https://discuss.elastic.co/u/dkyle)
#### Post date: [September 18, 2024, 8:40am UTC](https://discuss.elastic.co/t/no-ml-nodes-with-sufficient-capacity-for-trained-model-deployment/357517/6 "2024-09-18T08:40:27Z")

</div>

Depending on how you deployed your cluster you may have to set `xpack.ml.use_auto_machine_memory_percent` to true.

See [Machine learning settings in Elasticsearch | Elasticsearch Guide [8.15] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/ml-settings.html)

Either set it in the `elasticsearch.yml` file or via the cluster settings API

```auto
PUT _cluster/settings
{
  "persistent": {
    "xpack.ml.use_auto_machine_memory_percent": "true"
  }
}

```

---

<div class="post-metadata">

### Author: ![slytherin\_wyne](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/slytherin_wyne/32/137679_2.png) [@slytherin\_wyne](https://discuss.elastic.co/u/slytherin_wyne)
#### Post date: [September 18, 2024, 10:28am UTC](https://discuss.elastic.co/t/no-ml-nodes-with-sufficient-capacity-for-trained-model-deployment/357517/7 "2024-09-18T10:28:11Z")

</div>

thanks, I run the API code in kibana console and I got this

```auto
{
  "acknowledged": true,
  "persistent": {
    "xpack": {
      "ml": {
        "use_auto_machine_memory_percent": "true"
      }
    }
  },
  "transient": {}
}

```

But when i try to start deployment e5-small-multilingual ,it still get a statusCode 429.

---

<div class="post-metadata">

### Author: ![dkyle](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dkyle/32/59114_2.png) [@dkyle](https://discuss.elastic.co/u/dkyle)
#### Post date: [September 18, 2024, 2:07pm UTC](https://discuss.elastic.co/t/no-ml-nodes-with-sufficient-capacity-for-trained-model-deployment/357517/8 "2024-09-18T14:07:04Z")

</div>

Please can you paste the full error message here

---

<div class="post-metadata">

### Author: ![slytherin\_wyne](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/slytherin_wyne/32/137679_2.png) [@slytherin\_wyne](https://discuss.elastic.co/u/slytherin_wyne)
#### Post date: [September 19, 2024, 1:17am UTC](https://discuss.elastic.co/t/no-ml-nodes-with-sufficient-capacity-for-trained-model-deployment/357517/9 "2024-09-19T01:17:43Z")

</div>

That's the full error i got

```auto
{
  "statusCode": 429,
  "error": "Too Many Requests",
  "message": "[status_exception\n\tCaused by:\n\t\tillegal_state_exception: Could not start deployment because no suitable nodes were found, allocation explanation [none]\n\tRoot causes:\n\t\tstatus_exception: Could not start deployment because no ML nodes with sufficient capacity were found]: Could not start deployment because no ML nodes with sufficient capacity were found",
  "attributes": {
    "body": {
      "error": {
        "root_cause": [
          {
            "type": "status_exception",
            "reason": "Could not start deployment because no ML nodes with sufficient capacity were found"
          }
        ],
        "type": "status_exception",
        "reason": "Could not start deployment because no ML nodes with sufficient capacity were found",
        "caused_by": {
          "type": "illegal_state_exception",
          "reason": "Could not start deployment because no suitable nodes were found, allocation explanation [none]"
        }
      },
      "status": 429
    }
  }
}

```

I solve it by adding another node with 16GB RAM constrained .  
[https://www.elastic.co/guide/en/elasticsearch/reference/8.14/docker.html#\_add\_more\_nodes](https://www.elastic.co/guide/en/elasticsearch/reference/8.14/docker.html#_add_more_nodes)

---

<div class="post-metadata">

### Author: ![dkyle](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dkyle/32/59114_2.png) [@dkyle](https://discuss.elastic.co/u/dkyle)
#### Post date: [September 19, 2024, 9:18am UTC](https://discuss.elastic.co/t/no-ml-nodes-with-sufficient-capacity-for-trained-model-deployment/357517/10 "2024-09-19T09:18:23Z")

</div>

Thanks for the response @SlytherinWyne I'm glad you got this working.

In the error message I was expecting an explanation for why the model did not deploy but in your example it is `none`. This shouldn't happen, it's something for us to look into. Thanks the feedback.

> [@slytherin\_wyne](#):
>
> `allocation explanation [none]`
