ECE RAM to Storage Ratio with Restful API

[apaulsen (Agnes)]

Hi,

my question was partly answered already in ECE RAM to Storage Ratio

Adding this in advanced configuration via UI it works fine

"overrides": {
    "quota": {
    "fs_multiplier": 16
    }
    }
}

But trying to set the same via Restful API doesn't work - obviously I use the wrong syntax or the wrong place to define it.

My "data" overrides-setting is ignored.

This is my json file:

{
    "cluster_name" : "MYCluster",
    "plan" : {
        "elasticsearch" : {
          "version" : "6.6.0"
        },
        "cluster_topology" : [
          {
            "memory_per_node" : 2048,
            "node_count_per_zone" : 1,
            "node_type" : {
                "data" : true,
                "ingest" : true,
                "master" : true
            },
            "zone_count" : 1
          }
        ],
        "deployment_template": {
          "id": "c7159de3c438493b9925474390a8a6a0"
      }
    },
	"data" : {
	  "overrides": {
        "quota": {
          "fs_multiplier": 16
        }
      }
    }
}

TIA,
Agnes

[Alex_Piggott]

The way of changing the FS:RAM quota fromECE 2.x is to create an instance configuration and then use that - see https://www.elastic.co/guide/en/cloud-enterprise/current/ece-getting-started-templates.html

(To respond to the original question, for completeness -

There isn't a data section in the POST (create) cluster call.

Instead there is a settings object at the top level which contains all the parameters we allow users to change via the v1 API ... note this does not include the FS:RAM override, which should be handled as above, though it remains in our "internal API" which is what the advanced cluster data editor exposes)

[apaulsen (Agnes)]

So I'am a little bit confused.

1. I have a template which uses an instance configuration with wanted "storage_multiplier" 16.

2. I can reference this template in my cluster deployment json file.

3. During Cluster Deployment neither template nor instance configuration are used.

4. The resulting cluster has the default "storage_multiplier" 32.

5. So far only the advice to put the following lines into advanced configuration into "Data" Windows worked:

"overrides": {
"quota": {
"fs_multiplier": 16
}
}

But this is working with UI, not API.

I can't put these puzzle pieces together.

This is my instance configuration "myInstanceConfig", used by "myTemplate_1".

INSTANCE CONFIGURATION

 {
  "name": "myInstanceConfig",
  "description": "Ratio 1 GB Ram = 16 GB Storage",
  "storage_multiplier": 16.0,
  "discrete_sizes": {
    "sizes": [1024, 2048, 4096, 8192, 16384, 32768, 65536],
    "default_size": 1024,
    "resource": "memory"
  },
  "id": "46c871a88f4145b89e6f1d2602999354",
  "node_types": ["data", "master", "ingest"],
  "instance_type": "elasticsearch"
}

TEMPLATE

{
  "name": "myTemplate_1",
...
  "description": "1 Zone,  2GB RAM, xpack aktiviert",
  "id": "c7159de3c438493b9925474390a8a6a0",
  "cluster_template": {
    "plan": {
      "cluster_topology": [{
        "size": {
          "value": 1024,
          "resource": "memory"
        },
        "node_type": {
          "master": true,
          "data": false,
          "ingest": false
        },
        "instance_configuration_id": "master",
        "elasticsearch": {
          "user_settings_yaml": "# Note that the syntax for ..."
        },
        "zone_count": 1
      }, {
        "size": {
          "value": 2048,
          "resource": "memory"
        },
        "node_type": {
          "master": true,
          "data": true,
          "ingest": true
        },
        "instance_configuration_id": "46c871a88f4145b89e6f1d2602999354",
        "elasticsearch": {
          "user_settings_yaml": "# Note that the syntax for ..."
          "node_attributes": {

          }
        },
        "zone_count": 1
      }],
...	  
    "settings": {
      "dedicated_masters_threshold": 3
    },
...
  "system_owned": false
}

CLUSTER DEPLOYMENT

curl -k -X POST -u admin ... clusters/elasticsearch -H 'content-type: application/json' -d '{
"cluster_name" : "MyCluster",
  "plan" : {
  "elasticsearch" : {
      "version" : "6.6.0"
  },
  "cluster_topology" : [
   {
      "memory_per_node" : 2048,
      "node_count_per_zone" : 1,
      "node_type" : {
         "data" : true,
         "ingest" : true,
          "master" : true
    },
    "zone_count" : 1
    }
    ],
   "deployment_template": {
   "id": "c7159de3c438493b9925474390a8a6a0"
   }
  }
}'

[Alex_Piggott]

The templates are currently just so that the UI knows how to render the cluster create page

If creating clusters via the API then you just need to specify the instance_configuration_id ... ie POST the plan and settings sub-objects from template you created (ie under plan and settings in the top-level create model; you can also add kibana here as discussed in a different thread), and optionally adding the deployment_template sub-object (which helps the UI render the cluster but is otherwise unused)

(Or if not using templates at all, you can just add instance_configuration_id under the cluster_topology entries in a simpler plan like the one you had last)

Apologies for the confusion, you're not the only person to have asked this question when using the API, seems like we need to add some docs for this specific case

Alex

[apaulsen (Agnes)]

Works!

   "cluster_name" : "MYCluster",
   "plan" : {
   		"elasticsearch" : {
        	"version" : "6.6.0"
            },
      "cluster_topology" : [
        { 
            "memory_per_node" : 2048,
            "node_count_per_zone" : 1,
            "node_type" : {
               "data" : true,
               "ingest" : true,
               "master" : true
        },
        "instance_configuration_id": "46c871a88f4145b89e6f1d2602999354",
        "elasticsearch": {
          "user_settings_yaml": " ...",
          "node_attributes": {

          }
        },
        "zone_count": 1
         }
      ]
   }

Thank you!

[apaulsen (Agnes)]

Additional question:

How can I change the disk-ram-rate in my now existing deployment with this API-deployed cluster, referencing
"instance_configuration_id": "46c871a88f4145b89e6f1d2602999354".

I've tried to edit the instance configuration, but there is no opertunity to do that over UI:

TIA,

Best regards,
Agnes

[Alex_Piggott]

I believe you have to create a new instance config id and migrate over to that.

I hadn't realized that was the case until recently, but it appears some fields are non-editable with existing instance configs

[apaulsen (Agnes)]

So I would:

a) create a new instance configuration with id "xyz"
b) edit the Advanced cluster configuration --> Plan --> change the id to new instance_configuration_id": "xyz"

Is it as easy as this?

[Alex_Piggott]

For clusters that aren't tied to deployment templates, it's that simple

The UI uses deployment templates to decide which sliders to render, so if you just change the instance config id you might run into trouble there

I think (I haven't played with this yet) you'd have to do the following:

• Create the new instance config
• Create a new deployment template with the new instance config id
• Edit the cluster from the advanced view, changing the deployment template id and instance config id

I'm not sure what will happen if you just edited (would need to be via the API) the deployment template to include the new instance config id - I worry it would break the basic edit UI (but the advanced editor should be fine, then once you'd changed instance config id over the basic edit UI would work again)

[apaulsen (Agnes)]

@apaulsen

"cluster_name" : "MYCluster",

Perhaps it's not so complicated, because in my example cluster above I used only an instance configuration but not a template. So I would not have to change an instance id inside a template.
So it looks as if the remaining part is a) create a new instance config b) change the id in the advanced editor

[apaulsen (Agnes)]

Worked for me.