ECE Deployment within 2 Zones failed: "Plan change failed: Failed to validate cluster health/prerequisites."

Elastic Orchestration Elastic Cloud Enterprise (ECE)
1

Hi,

I've spin up new ECE cluster with 6 VMs (2 Controll Plane, 2 Allocators for Hot data and 2 Allocators for Warm data) as half of them are in Zone-1 others in Zone-2 repectively per roles.
I've setup tags for all my Allocators such as default:true/false, highstorage:true/false. Then I've modified respectively Instance configurations for data.default mathing only "Hot" allocators and data.highstorage to match tags only on "Warm" allcoators. Everytime I've tried to create new deployment it failed on Elasticsearch "Plan successful" step with "Plan change failed: Failed to validate cluster health/prerequisites.". Detailed error:

{
  "caused_by": "no.found.constructor.validation.ValidationException: 1. 
Could not ensure capacity allocation for [a56957bfd025492e930a309e7dc03e81] result
 was [{\"type\":\"capacity_by_zone\",\"allocated_instances\":3,
\"mappings\":[{\"key\":\"InstanceConfigurationId(data.default)\",\"mapping\":
{\"invalid\":false,\"logical_to_concrete\":{},\"concrete_to_logical\":{}}}],
\"required_capacity\":{},\"failed_instances\":[{\"instance_name\":
\"not-created-yet\", \"instance_configuration_id\":\"master\",
\"logical_zone_name\": \"tiebreaker\",\"instance_capacity\":1024,
\"type\":\"other\", \"rejections\":{}},{\"instance_name\":
\"not-created-yet\",\"instance_configuration_id\":\"data.default\",
\"logical_zone_name\":\"zone-0\",\"instance_capacity\":2048,\"type\":\"other\",
\"rejections\":{}},{\"instance_name\":\"not-created-yet\",\"instance_configuration_id\":\"data.default\",
\"logical_zone_name\":\"zone-1\",\"instance_capacity\":2048,
\"type\":\"other\",\"rejections\":{}}]}]"
}

I have more than 13GB+ RAM 200GB+ storage free on each VM and even when I configure the smallest ES within 2 Zones it failed, when setting only 1 Zone all good. I've tried to create custom Deployment Template providing edited Instance configurations for Hot and Warm data without success as well.

Will be happy to give some ideas or solution to my issue.
Thanks in advance,
Hristo.

2

Someone already face this issue?

3

I think what is happening is ECE is trying to inject a "tiebreaker" node with instance configuration type "master" since 2 node clusters are not a safe quorum, and it does not have anywhere to allocate this based on the template. High availability | Elastic Cloud Enterprise Reference [3.6] | Elastic

4

Hi Daniel,

I must agree that, this is the issue. Just to be clear, In order to have 2 zones ECE cluster and deployments, I need 3 zones, where last 3rd zone could container only 1 node for Master/tierbreaker to provide quorum if one zone failed? So every deployment with 2 Zones should have Allocators in 3 zones and that 3rd zone allocator should be tagged to be only "master" in Instance configuration? I don't get how to set deployment, 2 zones for data, and need more than 6 nodes to "activate" master node:

Master nodes
Dedicated master nodes will be automatically added once you reach 6 Elasticsearch nodes across all zones.

Thanks for your assistance!
Hristo.

5

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.