Evaluation Cluster - Unable to Scale to Second Zone

[gfidler (Gary Fidler)]

Summary

I am deploying a baseline cluster in AWS for evaluation. The software configuration and deployment on both nodes seems correct, according to the documentation. VM memory settings are default. (I had issues setting memory settings during install). Allocators are in 2 zones.

The second allocator currently has no nodes deployed.

I'm trying to deploy Elasticsearch and Kibana on the second allocator by setting 2 zones for these nodes in admin-console-elasticsearch. This configuration fails.

What am I missing?

Issue

Out of capacity

There is not enough capacity to complete your configuration change. The Allocators page indicates how much capacity is available.

To resolve this situation either add more capacity
or move nodes around

.
One of the following changes is likely to be the culprit:
Set size to 4 GB RAM
data.default
Set nodes per zone to 1
data.default
Set zones to 2
data.default

Hardware

2 x i3.4xlarge instances (High I/O, 16 CPU, 122GB Memory, 300GB Storage)

[Alex_Piggott]

Hi @gfidler

ECE actually requires 3 zones to provide HA (to get HA without any risk of split brain, you need >= 3 zones), with 1 and 2 zone deployments you can only spin up single-zone clusters.

Once you have a 3rd zone, you can spin up what are called "2-zone clusters" - but actually what that means is that you have 2 data zones, and a (small/cheap) "tiebreaker" node in the 3rd zone

A common thing that people do is install a smaller allocator in a 3rd zone and then use tags to ensure that only tiebreakers (and sometimes kibana nodes) go to that zone

[gfidler (Gary Fidler)]

Thanks Alex. I'll give that a shot and get back.

[gfidler (Gary Fidler)]

I have deployed new hardware. Two large boxes and one small.

• How do I configure the small host so that it becomes tiebreaker?
• What roles?
• Can I manually move nodes to different hosts?

My first attempt left one of my big hosts as the tiebreaker.

[Alex_Piggott]

It only needs the allocator role (if you have capacity you may want to add another director so that the ECE persistence store is also HA)

To configure the system to make zone 3 tiebreaker only, you would first go to the 3 allocators (platform > allocators), tag the smaller one as tiebreaker and the bigger two as data. Then go to the instance configs (platform > templates > instance configs) page and for each instance config (the names make it fairly clearly what instance types they represent) setup the filters