No ML nodes with sufficient capacity for trained model deployment

Elastic Stack Elasticsearch
1

Hi,
I'm new with elasticsearch. I currently have the trial period Platinum subscription, since I want to check if elasticsearch fits with what I want to do.

I want to test the elster_model_2 ML model. However, every time that I try to deploy it I get the following error:

Could not start deployment because no ML nodes with sufficient capacity were found

{
  "statusCode": 429,
  "error": "Too Many Requests",
  "message": "[status_exception\n\tCaused by:\n\t\tillegal_state_exception: Could not start deployment because no suitable nodes were found, allocation explanation [Could not assign (more) allocations on node [iYF2E0RcS8S5D_rUdpSBXA]. Reason: This node has insufficient available memory. Available memory for ML [322122547 (307.1mb)], free memory [322122547 (307.1mb)], estimated memory required for this model [469581194 (447.8mb)].|Could not assign (more) allocations on node [rmiiO1-LTtaZVPon05N39Q]. Reason: This node has insufficient available memory. Available memory for ML [322122547 (307.1mb)], free memory [322122547 (307.1mb)], estimated memory required for this model [469581194 (447.8mb)].|Could not assign (more) allocations on node [z9GMuSTwRuGuDGtwQI8BGQ]. Reason: This node has insufficient available memory. Available memory for ML [322122547 (307.1mb)], free memory [322122547 (307.1mb)], estimated memory required for this model [469581194 (447.8mb)].]\n\tRoot causes:\n\t\tstatus_exception: Could not start deployment because no ML nodes with sufficient capacity were found]: Could not start deployment because no ML nodes with sufficient capacity were found",
  "attributes": {
    "body": {
      "error": {
        "root_cause": [
          {
            "type": "status_exception",
            "reason": "Could not start deployment because no ML nodes with sufficient capacity were found"
          }
        ],
        "type": "status_exception",
        "reason": "Could not start deployment because no ML nodes with sufficient capacity were found",
        "caused_by": {
          "type": "illegal_state_exception",
          "reason": "Could not start deployment because no suitable nodes were found, allocation explanation [Could not assign (more) allocations on node [iYF2E0RcS8S5D_rUdpSBXA]. Reason: This node has insufficient available memory. Available memory for ML [322122547 (307.1mb)], free memory [322122547 (307.1mb)], estimated memory required for this model [469581194 (447.8mb)].|Could not assign (more) allocations on node [rmiiO1-LTtaZVPon05N39Q]. Reason: This node has insufficient available memory. Available memory for ML [322122547 (307.1mb)], free memory [322122547 (307.1mb)], estimated memory required for this model [469581194 (447.8mb)].|Could not assign (more) allocations on node [z9GMuSTwRuGuDGtwQI8BGQ]. Reason: This node has insufficient available memory. Available memory for ML [322122547 (307.1mb)], free memory [322122547 (307.1mb)], estimated memory required for this model [469581194 (447.8mb)].]"
        }
      },
      "status": 429
    }
  }
}

I curently have 3 ML nodes with 1GB total memory each (512 MB estimated available memory). Am I missing something? How can I change this? Would having more RAM solve this issue (I currently only have 32 GB)?

Thank you very much!

2

Hi @msola Welcome to the community!.

Well, in short, you don't have enough RAM on any one of the ml nodes to support the model.

In general, it's better to scale ml nodes vertically not horizontally so that you have more RAM in any one node.

If you are RAM constrained, it's probably a better idea to make one larger node than three smaller nodes.

1 GB nodes are very small in the ml world as each model it's going to take several hundred Mbs.

3

Thanks! The nodes were automatically set to 1GB, even if I only choose one node. How can I increase the size of one single node?

Does it mean that if I use a computer with a large RAM, the nodes will automatically have a larger memory?

4

@msola How did you deploy your cluster?

On Prem Self Managed Nodes, Dockers? K8s? Are they in containers?

Yes Elastic will set the RAM to 50% of the available system memory assuming that we have not changed configurations.
Did you set up specific ML Nodes?
You will need to share your configs otherwise we will just be guessing and answering a lot of partial questions :slight_smile: