Error during creating cluster

[weibin.wu]

Hi ECE:

During creating cluster, I got this error
Unexpected error during step: [ensure-s3-resources]: [java.lang.Exception: Unable to assign S3 resources: [No available resources.]

I did not setup any snapshot when create cluster. Why the cluster is trying to access S3?

[uricohen]

Hi @weibin.wu

Which version of ECE are you using?
Did you create a global snapshot repository or is this a fresh ECE installation?

Thanks,
Uri

[weibin.wu]

Hi
ECE version: 1.0 release
It is a new setup ECE
I create the cluster without inputing the cluster name.

Full error:

{
  "instance_count": 0,
  "instance_capacity": 0,
  "master_node_count": 0,
  "master_node_capacity": 0,
  "failover": {
    "skip_snapshot": false
  },
  "plan_id": "9a345144-d5e2-4062-a6f1-ac1a8f5e8037",
  "source": {
    "facilitator": "adminconsole",
    "action": "elasticsearch.stop-cluster",
    "date": "2017-06-09T07:16:11.559Z"
  },
  "strategy": {}
}
Detailed information for all steps
2017-06-09T07:16:11.604Z
Starting step: [resolve-node-type]:

clusterType=ElasticsearchClusterType
2017-06-09T07:16:11.607Z
Completed step: [resolve-node-type] with result:

None
This step took 1ms.
2017-06-09T07:16:11.608Z
Starting step: [plan-validator]:

cluster=ElasticsearchCluster(b0d429a3ff6c4a57bd22bbeec9ab61c3), stat=12884904386,12884904386,1496992571574,1496992571574,0,0,0,0,285,0,12884904386 }
2017-06-09T07:16:11.619Z
Completed step: [plan-validator] with result:

ElasticsearchPlan(Some(9a345144-d5e2-4062-a6f1-ac1a8f5e8037),0,0,1,0,0,Some(false),1024,false,1024 seconds,5 seconds,ElasticsearchPlanSection(None,List(),true,List(),List(),{},{},false,None),DefaultStrategy(),Failover(false,Map(),Map(),false,List(),true,false,3,None,false,None,false,false),TransientPlanSection(None),None,None,12884904386,12884904386,1496992571574,1496992571574,0,0,0,0,285,0,12884904386 )
This step took 10ms.
2017-06-09T07:16:11.619Z
Starting step: [validate-elasticsearch-plugin-versions]: []
2017-06-09T07:16:11.619Z
Completed step: [validate-elasticsearch-plugin-versions] with result:

()
This step took 0ms.
2017-06-09T07:16:11.620Z
Starting step: [ensure-s3-resources]: []
2017-06-09T07:16:11.626Z
Unexpected error during step: [ensure-s3-resources]: [java.lang.Exception: Unable to assign S3 resources: [No available resources.]
2017-06-09T07:16:11.627Z
Performing [0] rollback actions: [List()].
2017-06-09T07:16:11.629Z
Plan change failed: [An unexpected exception was raised]

[Alex_Piggott]

Hi @weibin.wu

This can happen if snapshots are enabled but a repository is not configured It's an odd state to be in though.

Under the advanced view (at the bottom of the Manage page for the cluster in question) in the there are 2 JSON text boxes, the first contains the plan, the second contains cluster metadata.

Can you paste the contents of the second? It should look something like:

{
    "name": "YOUR_NAME",
    "found-users": { /* various internal users, ignore */ },
   "snapshot": {
      "enabled": true|false,
      "supported": true|false,
      /* other things */
   },
   /* other things */
}

Also there is one other strange thing in your logs ... you were trying to create a cluster from scratch, but the plan says "action": "elasticsearch.stop-cluster" ... can you clarify how you were creating the cluster - were you just going to the cluster page, clicking create, and then filling in the fields?

Alex

[weibin.wu]

Hi Alex:

I confirm I dont any snapshot at "advanced configuration".

I think I found something. I create the cluster with 32GB / 1 TB size. However, each of the AZ only has 1TB storage maximum. So I am creating a cluster out of the capacity.

After creation, I dont see the cluster appear in Platform tab.
So I deleted it. And during deleting, this error comes out.

Now the cluster becomes undeletable.

Is there any way to resolve this problem instead of re-install the whole ECE?

[Alex_Piggott]

Hi @weibin.wu

Can you paste the following into the "Data" block in the aforementioned "advanced cluster configuration" page.

  "snapshot": {
    "enabled": false,
    "suspended": {}
  },

"Save" that, then the "Shutdown" button should work (and then the "Delete" button).

What's happening is that for some reason (we can investigate that once this issue is fixed) the snapshot block is not getting written to the cluster metadata, and it defaults to true.

[weibin.wu]

Thanks Alex:

It works. Now i can stop and delete that cluster.

[Alex_Piggott]

Excellent, I'm glad. Now ... if you create another cluster (whether it succeeds or fails), you should see that it always creates that snapshot: { enabled: false } object in the cluster metadata? (ie what you ran into was a one-off?)

I have the UI team looking into why that might have happened, but it would be useful to confirm that it was intermittent and not something specific to your set up

Thanks for all your help ironing out these teething troubles!

Alex