ECE expand storage query

Elastic Orchestration Elastic Cloud Enterprise (ECE)
1

Hi All,
I use ESX VM and on top of it I run containers in my poc. I configured customized instance/template to adjust ram:storage ratio mainly to allocate/use only needed storage. Say for ex:- I allocate 1:100
Since the VM has limited storage initially and will be extended on VM as and when the need arises.
So to handle such situation I have 2 scenario

A) I have more RAM with less storage so that in future I only need to add storage as needed
B) I have less ram initially with less storage to start with as there is less data to process.

Now the question is in future when more data lands into the cluster

1) Can I extend only the storage on specific roles like HOT or WARM containers without downtime and without increasing RAM?
2) Can I increase RAM + Storage on specific roles like HOT or WARM containers without downtime?
3) I hope the quotas for RAM & Storage can be configured at allocator level or at container level?
4) How do I know from CLI whether specific container storage has been increased from previously configured size? 
5) I hope the storage on esx VM can be extended to existing mounted partition on VM where all containers are located without downtime
2

For disk - ECE actually has no knowledge of the disk capacity ... it just assumes there is sufficient disk for the given RAM multiplied by the storage:RAM ratio. So upping the disk capacity is just an OS activity.

Increasing RAM on an already running allocator is currently a bit painful.

There is an API but it has a "bug" that it doesn't recalculate the associated quotas; otherwise the recommendation is to reinstall with the new value. (I think there is a workaround using undocumented APIs, that in some cases support has worked with customers on)

3

Thanks for your quick revert Alex.

Basically I did checked, the ram:storage ratio is allocated as per the multiplier set by default and with standard settings we cannot change that unless as you mentioned some undocumented way. Hence wanted to clarify on below

Whether the above scenario can be achieved or not?

I tried this but was not working as no option available that I can see, may be someone else can guide if this is possible

It does increases but unsure about what will happen to the incoming log data while the change is in progress, as the change is in-progress

Not sure about this, can someone throw some light on this?

4

I think I got confused by whether you were talking about adding RAM to the VM or the cluster.

Let me try to give a few concrete use cases:

You have a 64GB RAM VM with 40GB assigned to the allocator. You have a single-node 20GB running on that VM cluster with 5:1 disk:ram (ECE doesn't care about how much disk you actually have, I'm going to assume you have enough, or add more if not following the OS/VM instructions)

1] You want to increase the cluster size to 40GB - you simply drag the slider for the given instance config (ie you can have different values for hot/warm). If it's an HA cluster there will be no downtime (possibly a few 50xs which the client should retry on). If it's a non-HA cluster you will get a few minutes of 50xs

2] You want to keep the cluster size as 20GB but increase the storage:RAM ratio to 10:1 ... this is slightly less standard (since the storage:RAM ratio is a function of application so won't normally change much), but you can do it by adding a new instance config and then using the API/advanced editor to change the instance config id (so is applicable to just one of hot/warm). Again, it will apply a rolling plan so the same availability principles apply

3] You want to add more RAM to your VM (eg upping it to 128GB) and then increase your cluster size to 60GB - this is the "not formally support except by wipe/reinstall" case I mentioned in the previous reply, which isn't ideal currently.

I think this covers everything except

  1. I hope the quotas for RAM & Storage can be configured at allocator level or at container level?

Which I'm not sure exactly what you mean. Each allocator VM can run multiple containers (eg ES nodes), so eg a 64GB allocator with 40GB allocatable capacity would run 2x 20GB nodes, 1x 40, 4x 10 etc. The allocator RAM and container RAM are independent except that the sum of the container RAMs on a given allocator cannot exceed the RAM of the allocator

5

I think I also got bit confused now :slight_smile: let me try to put up my case may be better this time, sorry if I am repeating

Scenario 1.A - More RAM with less storage:

I have 3 allocators each having 128 RAM = 384 GB RAM in total and 3 TB storage for each allocator = 9 TB total storage. 
I run 6 warm containers on these allocators. The RAM:Storage ratio set to 1:100 in the instance config and I run hot-warm.  
I run 2 containers per VM. Each warm container is allocated 64 GB RAM. 
The storage as you mentioned the ECE doesn't care so using it as per availability

Question is -

can I allocate 64 GB RAM for each warm containers though the storage per container I get is only 1.5 TB per container? 
because on the other hand the ram:storage ratio is 1:100 will that work?

Scenario 1.B: Expanding Storage

Consider more data getting ingested and we increase from 3TB to 15 TB per allocator  that is each container on that allocator will get 7.5 TB. No RAM increase activity.

Questions is - Can I increase storage alone from ECE?

or there is no need for such activity to carry as the ECE will automatically detect the added storage into allocator VM and the containers can use that added space with no additional efforts

Scenario 2:- Less RAM with less storage

I have 3 allocators with 128 RAM each = 384 GB RAM in total and 3 TB storage for each allocator = 9 TB total storage. 
I run 6 warm containers on these allocators. The RAM:Storage set to 1:100 and I run hot-warm.  
Each warm container is allocated 32 GB RAM. That means still 32 GB is not allocated to containers and available in allocator

After few days the demand for processing increases and hence requirement of increased RAM comes up

So the Question is :-

1. Can I increase only RAM of warm containers without downtime and without increase in storage? earlier the container had 32GB RAM with 1.5 TB storage. 
2. If yes for above then hope it will not look for available storage is sufficient or not as per the ram:storage ratio to increase it proportionally

Scenario 3:- Allocate odd number of RAM per container

I have 3 allocators with 128 RAM each = 384 GB RAM in total and 3 TB storage for each allocator = 9 TB total storage. 
I run 6 warm containers on these allocators. The RAM:Storage set to 1:100.

Question:

1. Can I allocate 38 GB RAM to warm containers? 
Asking this question because at higher side I do see only 32 and 64 GB RAM option that can be allocated and I am not able to add the custom odd ram figure between 32-64 GB in the instance configuration.

2. If above is possible then how to implement this?

Quota question:

From above example, I want to set quota for disk storage utilization per container wise, so is it possible? if yes then how to implement Quota for storage at container level?

Hope I was able to elaborate my questions now

6

can I allocate 64 GB RAM for each warm containers though the storage per container I get is only 1.5 TB per container?
because on the other hand the ram:storage ratio is 1:100 will that work?

You will be allowed to allocate the 64GB RAM - the only proviso if you don't have sufficient storage is that the allocator will crash horribly if you run out of disk space (which you can do because as far as ECE is concerned each node has 6TB)

You can probably easily mitigate this by using disk watermarks in ES though

Questions is - Can I increase storage alone from ECE?

First you need to add disk space to the OS. I don't know whether in your envionment that requires bringing the allocator down. Once the disk space is visible by the OS, ECE will happily use it (ie the limit is per container and based on RAM * disk:RAM ratio)

Can I increase only RAM of warm containers without downtime

If your cluster is "high availability" (eg each allocator is a zone, or each shard has >1 replica) then you can increase the RAM without downtime

If yes for above then hope it will not look for available storage is sufficient or not as per the ram:storage ratio to increase it proportionally

It will not check storage (ECE only ever checks per container storage limit, it knows nothing about the actual total per allocator storage - for good or bad!)

Can I allocate 38 GB RAM to warm containers?

You can only use one of the "increment" sizes in the instance config being used in the deployment template. So this question is more related to the other thread of how to build instance configs using the API.

But basically you can decide to have [0, 10, 20, 30, 40, 50, 60]and then those are allowable per-node memories (instead of 32 or 64)

So you do need to decide what the allowable values are in advance

7

How to get this change done? any guidelines for it? I see only [1,2,4,...32,64]

8

I think the recommended approach is to get an instance config the way you want from what is achievable from the UI, then GET that instance config, then edit the fields you want to be different, change the id/name, then POST it

(the version of the UI I have has a handy "equivalent API request" button that makes this step more elegant, but I think that might not be released in ECE yet ... here's what I get when I edit one of our production warm instances)

(note the "weird" instance sizes we used to optimize bin packing into our allocator)

POST /api/v1/regions/test-region/platform/configuration/instances
{
  "name": "aws.data.highstorage.d2",
  "description": "A storage optimized Elasticsearch instance running on an AWS d2.",
  "storage_multiplier": 160,
  "instance_type": "elasticsearch",
  "node_types": [
    "master",
    "data",
    "ingest"
  ],
  "discrete_sizes": {
    "sizes": [
      2048,
      4096,
      8192,
      15360,
      29696,
      59392
    ],
    "default_size": 4096,
    "resource": "memory"
  },
  "allocator_filter": {
    "bool": {
      "must": [
        {
          "bool": {
            "must": [
              {
                "nested": {
                  "path": "metadata",
                  "query": {
                    "bool": {
                      "must": [
                        {
                          "term": {
                            "metadata.key": {
                              "value": "instanceFamily"
                            }
                          }
                        },
                        {
                          "term": {
                            "metadata.value.keyword": {
                              "value": "d2"
                            }
                          }
                        }
                      ]
                    }
                  }
                }
              }
            ]
          }
        }
      ]
    }
  }
}
9

Got error

  "errors": [{
    "code": "root.unsupported_content_type",
    "message": "The supplied content-type is not supported. Expected 'application/json'"
  }]

this is the query, not sure what is wrong, its the same and I modified only few items

curl -XPOST -u admin:password http://server:port/api/v1/platform/configuration/instances -H 'content-type: applicaiton/json' -d'
{
  "name": "warm",
  "description": "A storage optimized Elasticsearch instance running on an AWS d2.",
  "storage_multiplier": 7,
  "instance_type": "elasticsearch",
  "node_types": [
    "data",
    ],
  "discrete_sizes": {
    "sizes": [
      2048, 4096, 8192, 15360, 29696, 59392
    ],
    "default_size": 4096,
    "resource": "memory"
  },
  "allocator_filter": {
    "bool": {
      "must": [
        {
          "bool": {
            "must": [
              {
                "nested": {
                  "path": "metadata",
                  "query": {
                    "bool": {
                      "must": [
                        {
                          "term": {
                            "metadata.key": {
                              "value": "instanceFamily"
                            }
                          }
                        },
                        {
                          "term": {
                            "metadata.value.keyword": {
                              "value": "d2"
                            }
                          }
                        }
                      ]
                    }
                  }
                }
              }
            ]
          }
        }
      ]
    }
  }
}'
10

I think you misspelled application:

curl -XPOST -u admin:password http://server:port/api/v1/platform/configuration/instances -H 'content-type: applicaiton/json' -d'
^^^^^^^^^^^^^^^^^^^^
here?

11

Yes thanks but still errors out

  "errors": [{
    "code": "root.malformed_request_content",
    "message": "General malformed error: [Unexpected character ']' at input index 203 (line 9, position 5), expected JSON Value:\n    ],\n    ^\n] ([Some(spray.json.JsonParser$ParsingException)])"
  }]
12

You should probably run that JSON object through jsonlint.com or similar, eg I didn't examine it exhaustively but I saw:

  "node_types": [
    "data",
    ],

That trailing , after "data" isn't valid JSON

13

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.