500: An internal server error

Elastic Orchestration Elastic Cloud Enterprise (ECE)
1

I've tried to change the cluster memory from 1GB to 2GB from ECE console. Now no cluster(including logging and metrics cluster) is coming up and showing error "500: An internal server error ". I've enough memory left, about 4GB.

I'm facing downtime.
how should i proceed for this?

Thanks
Annu

2

I moved the question to #cloud-enterprise

3

Hi @annu

Can you provide a bit more detail on what happened exactly?

Eg it sounds like ...

  • You had an ECE platform (what configuration, eg how many allocators etc, how much RAM etc)?
  • It had some additional clusters in it that you'd created (how many/what sizes)
  • You increased the size of one of them from 1GB to 2GB

At which point the UI stopped working?

4

@annu

It sounds like the adminconsole cluster (or possibly the whole allocator hosting it) went down, which takes the UI down with it

You can check what's going on without the UI, using the API (docs: https://www.elastic.co/guide/en/cloud-enterprise/current/ece-api-reference.html)

eg curl -u root:PASSWORD 'HOST:12400/clusters/elasticsearch' (use readonly and its password to generate a sanitized list) will list the clusters together with some basic health information, /platform/infrastructure/allocators will list the health of the allocators etc

5

Hi

I was using 32GB disk and 1GB memory for my cluster(i created). Due to the increased memory pressure, i changed the memory from 1GB to 2GB. While that migration was happening, my UI went down and now I'm getting "Fetching clusters failed" and "service unavailable" errors when i login to my ELK.

And i'm getting "{"ok":false,"message":"The requested cluster is currently split"}" when i try to hit my elastic search endpoint.

thanks
Annu

6

elasticsearch
elasticsearch.png1019×419 4.42 KB
cluster
cluster.png1593×399 12.6 KB

7

Thanks for the details, seems like 2 things have happened:

  • When updating an overloaded cluster you got a split brain situation - this has started happening with a bit of regularity, and once we get the platform back I will provide some strategies for dealing with it
  • The split brain has somehow taken down the administrative cluster that the UI uses to display the clusters ... this I haven't seen before

Can you run a few API commands as the user readonly (which ensures you don't leak any secrets)

  • curl -u readonly:PASSWORD "http://HOST:12400/api/v1/clusters/elasticsearch" - returns a list of all the clusters and their health
  • curl -u readonly:PASSWORD "http://HOST:12400/api/v1/clusters/elasticsearch/XXX" where XXX is the adminconsole cluster return from the previous call
  • curl -u readonly:PASSWORD "http://HOST:12400/api/v1/clusters/elasticsearch/YYY/plan/activity" where YYY is the cluster that you upgraded and split brained
  • curl -u readonly:PASSWORD "http://HOST:12400/api/v1/platform/infrastructure/allocators" - provides information about the health of the allocators that run the clusters

(API reference: https://www.elastic.co/guide/en/cloud-enterprise/current/ece-api-reference.html)

8

i'm not able to login to Elastic cloud.
when i tried : curl -u readonly:PASSWORD "http://HOST:12400/api/v1/clusters/elasticsearch"
i'm getting authentication error
{
"errors": [{
"code": "root.unauthenticated",
"message": "The supplied authentication is invalid"
}]
}

9

@annu - sorry the PASSWORD i meant was the password corresponding to the user, eg it was displayed when you installed it:

Admin Console Details
URL                 :  http://HOST:12400
Root username       :  root
Root password       :  Cus8qwjzlSI22Tm73a9hITNgWaf304MIACsTSjdzufh
Read-only username  :  readonly
Read-only password  :  SwO7kMNCHeUsUGVoouvZbOXt0OgmLEe79NdF9LAYFgc

if you didn't take a note of it at the time, not to worry, all the credentials are kept /mnt/data/elastic/bootstrap-state/bootstrap-secrets.json (may vary if you installed in a different root directory)

10

Hi @Alex_Piggott
i give the same password on my read-only field.

thanks
Annu

11

Does it work with root?

12

@Alex_Piggott

I'm not getting any output when i run curl -u readonly:PASSWORD "http://HOST:12400/api/v1/clusters/elasticsearch" this command.

i tried using with root password also. but no result.

13

Try adding -vvv in there, that will give a lot more details about the failure, eg curl -vvv- u <etc etc>

14

@annu I believe we have a fix for your issue here Kibana Instances "Unable to Connect to Server"

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.