ECE Deployment within 2 Zones failed: "Plan change failed: Failed to validate cluster health/prerequisites."

[Icaka (Hristo Stoychev)]

Hi,

I've spin up new ECE cluster with 6 VMs (2 Controll Plane, 2 Allocators for Hot data and 2 Allocators for Warm data) as half of them are in Zone-1 others in Zone-2 repectively per roles.
I've setup tags for all my Allocators such as default:true/false, highstorage:true/false. Then I've modified respectively Instance configurations for data.default mathing only "Hot" allocators and data.highstorage to match tags only on "Warm" allcoators. Everytime I've tried to create new deployment it failed on Elasticsearch "Plan successful" step with "Plan change failed: Failed to validate cluster health/prerequisites.". Detailed error:

{
  "caused_by": "no.found.constructor.validation.ValidationException: 1. 
Could not ensure capacity allocation for [a56957bfd025492e930a309e7dc03e81] result
 was [{\"type\":\"capacity_by_zone\",\"allocated_instances\":3,
\"mappings\":[{\"key\":\"InstanceConfigurationId(data.default)\",\"mapping\":
{\"invalid\":false,\"logical_to_concrete\":{},\"concrete_to_logical\":{}}}],
\"required_capacity\":{},\"failed_instances\":[{\"instance_name\":
\"not-created-yet\", \"instance_configuration_id\":\"master\",
\"logical_zone_name\": \"tiebreaker\",\"instance_capacity\":1024,
\"type\":\"other\", \"rejections\":{}},{\"instance_name\":
\"not-created-yet\",\"instance_configuration_id\":\"data.default\",
\"logical_zone_name\":\"zone-0\",\"instance_capacity\":2048,\"type\":\"other\",
\"rejections\":{}},{\"instance_name\":\"not-created-yet\",\"instance_configuration_id\":\"data.default\",
\"logical_zone_name\":\"zone-1\",\"instance_capacity\":2048,
\"type\":\"other\",\"rejections\":{}}]}]"
}

I have more than 13GB+ RAM 200GB+ storage free on each VM and even when I configure the smallest ES within 2 Zones it failed, when setting only 1 Zone all good. I've tried to create custom Deployment Template providing edited Instance configurations for Hot and Warm data without success as well.

Will be happy to give some ideas or solution to my issue.
Thanks in advance,
Hristo.

[Icaka (Hristo Stoychev)]

Someone already face this issue?

[Daniel_Battaglia]

@Icaka

"instance_configuration_id\":\"master\",
\"logical_zone_name\": \"tiebreaker\"

I think what is happening is ECE is trying to inject a "tiebreaker" node with instance configuration type "master" since 2 node clusters are not a safe quorum, and it does not have anywhere to allocate this based on the template. High availability | Elastic Cloud Enterprise Reference [3.6] | Elastic

[Icaka (Hristo Stoychev)]

Hi Daniel,

I must agree that, this is the issue. Just to be clear, In order to have 2 zones ECE cluster and deployments, I need 3 zones, where last 3rd zone could container only 1 node for Master/tierbreaker to provide quorum if one zone failed? So every deployment with 2 Zones should have Allocators in 3 zones and that 3rd zone allocator should be tagged to be only "master" in Instance configuration? I don't get how to set deployment, 2 zones for data, and need more than 6 nodes to "activate" master node:

Master nodes
Dedicated master nodes will be automatically added once you reach 6 Elasticsearch nodes across all zones.

Thanks for your assistance!
Hristo.