ECE installation failure - 1.1.5

Elastic Orchestration Elastic Cloud Enterprise (ECE)
1

Hi,

I've been trying to install ece to test but i'm getting the following output and can't see any obvious problem to try to fix it. This is an offline installation so i have created an internal docker registry with the ECE image and 6.4 images for ES and Kibana.

...

-bash-4.2$ bash ./elastic-cloud-enterprise.sh install --docker-registry host2.internal.com:5000 --host-storage-path /local/1/elastic-cloud --debug

  • Creating Admin Console Elasticsearch backend {}
  • Unhandled error. {}
    -- An error has occurred in bootstrap process. Please examine logs --
    java.util.concurrent.TimeoutException: Futures timed out after [30 minutes]
    at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:223)
    at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:227)
    at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:190)
    at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
    at scala.concurrent.Await$.result(package.scala:190)
    at no.found.bootstrap.BootstrapInitial.bootstrapServiceLayer(BootstrapInitial.scala:880)
    at no.found.bootstrap.BootstrapInitial.bootstrap(BootstrapInitial.scala:650)
    at no.found.bootstrap.BootstrapInitial$.delayedEndpoint$no$found$bootstrap$BootstrapInitial$1(BootstrapInitial.scala:1215)
    at no.found.bootstrap.BootstrapInitial$delayedInit$body.apply(BootstrapInitial.scala:1209)
    at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
    at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
    at scala.App$$anonfun$main$1.apply(App.scala:76)
    at scala.App$$anonfun$main$1.apply(App.scala:76)
    at scala.collection.immutable.List.foreach(List.scala:392)
    at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
    at scala.App$class.main(App.scala:76)
    at no.found.bootstrap.BootstrapInitial$.main(BootstrapInitial.scala:1209)
    at no.found.bootstrap.BootstrapInitial.main(BootstrapInitial.scala)
2

Hi,
I am also facing same issue for the same version of ECE
The error is below:

  • [2018-09-18 02:19:21,374][INFO ][no.found.bootstrap.BootstrapInitial] Applying Admin Console Elasticsearch index templates {}
  • [2018-09-18 02:19:21,677][INFO ][no.found.bootstrap.ServiceLayerBootstrap] Waiting for [apply-elasticsearch-template] to complete. Retrying every [1 second] (cause: [java.lang.Exception: 502 Bad Gateway: HttpEntity(application/json; charset=UTF-8,{"ok":false,"message":"The requested cluster is currently unavailable"})]) {}
  • [2018-09-18 02:29:21,379][ERROR][no.found.bootstrap.BootstrapInitial$] Unhandled error. {}
    -- An error has occurred in bootstrap process. Please examine logs --
  • java.util.concurrent.TimeoutException: Futures timed out after [600000 milliseconds]
  • at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:223)
  • at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:227)
  • at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:190)
  • at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
  • at scala.concurrent.Await$.result(package.scala:190)
  • at no.found.bootstrap.BootstrapInitial.bootstrapServiceLayer(BootstrapInitial.scala:892)
  • at no.found.bootstrap.BootstrapInitial.bootstrap(BootstrapInitial.scala:650)
  • at no.found.bootstrap.BootstrapInitial$.delayedEndpoint$no$found$bootstrap$BootstrapInitial$1(BootstrapInitial.scala:1215)
  • at no.found.bootstrap.BootstrapInitial$delayedInit$body.apply(BootstrapInitial.scala:1209)
  • at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
  • at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
  • at scala.App$$anonfun$main$1.apply(App.scala:76)
  • at scala.App$$anonfun$main$1.apply(App.scala:76)
  • at scala.collection.immutable.List.foreach(List.scala:392)
  • at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
  • at scala.App$class.main(App.scala:76)
  • at no.found.bootstrap.BootstrapInitial$.main(BootstrapInitial.scala:1209)
  • at no.found.bootstrap.BootstrapInitial.main(BootstrapInitial.scala)
  • [2018-09-18 02:29:21,476][INFO ][no.found.bootstrap.ServiceLayerBootstrap] Waiting for [apply-elasticsearch-template] to complete. Retrying every [1 second] (cause: [akka.pattern.AskTimeoutException: Recipient[Actor[akka://default/user/IO-HTTP#-329960940]] had already been terminated. Sender[null] sent the message of type "spray.http.HttpRequest".]) {}
  Errors have caused Elastic Cloud Enterprise installation to fail - Please check logs 
  Node type - initial

Can someone help me to figure this out
Thanks for your help

3

Is there any solution for this? I am experiencing the same problem when installing ECE 1.1.5 with internet access.

Applying Admin Console Elasticsearch index templates {}
[2018-09-23 22:18:03,029][INFO ][no.found.bootstrap.ServiceLayerBootstrap] Waiting for [apply-elasticsearch-template] to complete. Retrying every [1 second] (cause: [java.lang.Exception: 502 Bad Gateway: HttpEntity(application/json; charset=UTF-8,{"ok":false,"message":"The requested cluster is currently unavailable"})]) {}
[2018-09-23 22:28:02,995][ERROR][no.found.bootstrap.BootstrapInitial$] Unhandled error. {}
java.util.concurrent.TimeoutException: Futures timed out after [600000 milliseconds]
       at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:223)
       at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:227)
       at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:190)
       at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
       at scala.concurrent.Await$.result(package.scala:190)
       at no.found.bootstrap.BootstrapInitial.bootstrapServiceLayer(BootstrapInitial.scala:892)
       at no.found.bootstrap.BootstrapInitial.bootstrap(BootstrapInitial.scala:650)
       at no.found.bootstrap.BootstrapInitial$.delayedEndpoint$no$found$bootstrap$BootstrapInitial$1(BootstrapInitial.scala:1215)
       at no.found.bootstrap.BootstrapInitial$delayedInit$body.apply(BootstrapInitial.scala:1209)
       at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
       at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
       at scala.App$$anonfun$main$1.apply(App.scala:76)
       at scala.App$$anonfun$main$1.apply(App.scala:76)
       at scala.collection.immutable.List.foreach(List.scala:392)
       at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
       at scala.App$class.main(App.scala:76)
       at no.found.bootstrap.BootstrapInitial$.main(BootstrapInitial.scala:1209)
       at no.found.bootstrap.BootstrapInitial.main(BootstrapInitial.scala)
[2018-09-23 22:28:03,023][INFO ][no.found.bootstrap.ServiceLayerBootstrap] Waiting for [apply-elasticsearch-template] to complete. Retrying every [1 second] (cause: [akka.pattern.AskTimeoutException: Recipient[Actor[akka://default/user/IO-HTTP#-2009195629]] had already been terminated. Sender[null] sent the message of type "spray.http.HttpRequest".]) {}
4

@darren.skinner

Sorry for the delay in responding - I believe to get the ES and Kibana images installed using "--docker-registry" may not be sufficient

The allocator logs (eg /mnt/data/elastic/:ALLOCATOR_ID/services/allocator/logs should tell you what image it's trying to pull so you can figure out where exactly to put the image that are downloadable from here: ... docs for offline install are here: https://www.elastic.co/guide/en/cloud-enterprise/current/ece-installing-offline.html#ece-installing-offline-images)

@sim , @martinmarialouis

What does docker ps return?

It seems likely that the cluster has spun up but is failing for some reason

The ES logs should be present in somewhere like /mnt/data/elastic/ALLOCATOR_ID/services/allocators/containers/elasticsearch/CLUSTER_ID/logs, which might give some indication

Alex

5

Thanks for responding @Alex_Piggott
No log directory exists in the /mnt/data/elastic/ALLOCATOR_ID/services/allocators/containers/elasticsearch/CLUSTER_ID, but there is a log in /mnt/data/elastic/:ALLOCATOR_ID/services/allocator/logs. I've included a snipit of the log file below. The last few lines repeate until it dies.

[2018-09-24 14:55:00,381][INFO ][no.found.runner.allocation.elasticsearch.ElasticsearchDockerContainerManager] Deleting container {"ec_container_kind":"elasticsearch","ec_container_group":"252e3238734343dbab69f5b3bb27a6e3","ec_container_name":"instance-0000000001"}
[2018-09-24 14:55:00,381][INFO ][no.found.docker.DockerContainerManager] Destroying container [fac-252e3238734343dbab69f5b3bb27a6e3-instance-0000000001] {"ec_container_kind":"elasticsearch","ec_container_group":"252e3238734343dbab69f5b3bb27a6e3","ec_container_name":"instance-0000000001"}
[2018-09-24 14:55:00,424][INFO ][no.found.runner.allocation.elasticsearch.ElasticsearchDockerContainerManager] Tearing down filesystem {"ec_container_kind":"elasticsearch","ec_container_group":"252e3238734343dbab69f5b3bb27a6e3","ec_container_name":"instance-0000000001"}
[2018-09-24 14:55:00,425][INFO ][no.found.runner.managers.XFSQuotaManager] Removing quota from [XFSProjectName(es,252e3238734343dbab69f5b3bb27a6e3,instance-0000000001)] {"ec_container_kind":"elasticsearch","ec_container_group":"252e3238734343dbab69f5b3bb27a6e3","ec_container_name":"instance-0000000001"}
[2018-09-24 14:55:00,429][WARN ][no.found.runner.managers.XFSQuotaManager] Command [[sudo, -n, xfs_quota, -x, -c, project -C -p /mnt/data/elastic/10.94.52.10/services/allocator/containers/elasticsearch/252e3238734343dbab69f5b3bb27a6e3/instance-0000000001/data 13008, /mnt/data]] returned status code [1]] with output: [E: [xfs_quota: cannot setup path for project dir /mnt/data/elastic/10.94.52.10/services/allocator/containers/elasticsearch/252e3238734343dbab69f5b3bb27a6e3/instance-0000000001/data: No such file or directory]] {}
[2018-09-24 14:55:00,432][WARN ][no.found.runner.managers.XFSQuotaManager] Command [[sudo, -n, xfs_quota, -x, -c, project -C -p /mnt/data/elastic/10.94.52.10/services/allocator/containers/elasticsearch/252e3238734343dbab69f5b3bb27a6e3/instance-0000000001/logs 13008, /mnt/data]] returned status code [1]] with output: [E: [xfs_quota: cannot setup path for project dir /mnt/data/elastic/10.94.52.10/services/allocator/containers/elasticsearch/252e3238734343dbab69f5b3bb27a6e3/instance-0000000001/data: No such file or directory], E: [xfs_quota: cannot setup path for project dir /mnt/data/elastic/10.94.52.10/services/allocator/containers/elasticsearch/252e3238734343dbab69f5b3bb27a6e3/instance-0000000001/logs: No such file or directory]] {}
[2018-09-24 14:55:00,435][WARN ][no.found.runner.managers.XFSQuotaManager] Command [[sudo, -n, xfs_quota, -x, -c, limit -p bhard=0 rtbhard=0 13008, /mnt/data]] returned status code [1]] with output: [E: [xfs_quota: cannot setup path for project dir /mnt/data/elastic/10.94.52.10/services/allocator/containers/elasticsearch/252e3238734343dbab69f5b3bb27a6e3/instance-0000000001/data: No such file or directory], E: [xfs_quota: cannot setup path for project dir /mnt/data/elastic/10.94.52.10/services/allocator/containers/elasticsearch/252e3238734343dbab69f5b3bb27a6e3/instance-0000000001/logs: No such file or directory], E: [xfs_quota: cannot set limits: Function not implemented]] {}
[2018-09-24 14:55:00,436][INFO ][no.found.runner.allocation.elasticsearch.ElasticsearchDockerContainerManager] Releasing network resources {"ec_container_kind":"elasticsearch","ec_container_group":"252e3238734343dbab69f5b3bb27a6e3","ec_container_name":"instance-0000000001"}
[2018-09-24 14:55:00,470][INFO ][no.found.runner.allocation.elasticsearch.ElasticsearchDockerContainerManager] Deallocated container {"ec_container_kind":"elasticsearch","ec_container_group":"252e3238734343dbab69f5b3bb27a6e3","ec_container_name":"instance-0000000001"}
[2018-09-24 14:55:00,470][INFO ][no.found.runner.allocation.elasticsearch.ElasticsearchDockerContainerManager] Telling [Actor[akka://default/user/$b/allocations#790836080]] about a successful deallocation {"ec_container_kind":"elasticsearch","ec_container_group":"252e3238734343dbab69f5b3bb27a6e3","ec_container_name":"instance-0000000001"}
[2018-09-24 14:55:00,470][INFO ][no.found.runner.allocation.AllocatorAllocationsManager] Stopping allocation: [Actor[akka://default/user/$b/allocations/elasticsearch-252e3238734343dbab69f5b3bb27a6e3-instance-0000000001-2#1426336881]] {}
[2018-09-24 14:55:00,471][INFO ][no.found.runner.allocation.elasticsearch.clustertask.ClusterTaskScheduler] [Actor[akka://default/user/$b/allocations/elasticsearch-252e3238734343dbab69f5b3bb27a6e3-instance-0000000001-2/scheduler#-1342683483]] Stopping scheduling of tasks {"ec_container_kind":"elasticsearch","ec_container_group":"252e3238734343dbab69f5b3bb27a6e3","ec_container_name":"instance-0000000001"}
[2018-09-24 14:55:00,471][INFO ][no.found.runner.allocation.elasticsearch.clustertask.ClusterTaskScheduler] [Actor[akka://default/user/$b/allocations/elasticsearch-252e3238734343dbab69f5b3bb27a6e3-instance-0000000001-2/scheduler#-1342683483]] Closing leader latch {"ec_container_kind":"elasticsearch","ec_container_group":"252e3238734343dbab69f5b3bb27a6e3","ec_container_name":"instance-0000000001"}
[2018-09-24 14:55:04,439][WARN ][no.found.runner.managers.XFSQuotaWatcher] Command [[sudo, -n, xfs_quota, -x, -c, report -p -n -N]] returned status code [0]] with output: [E: XFS_GETQUOTA: Invalid argument] {}
[2018-09-24 14:55:34,460][WARN ][no.found.runner.managers.XFSQuotaWatcher] Command [[sudo, -n, xfs_quota, -x, -c, report -p -n -N]] returned status code [0]] with output: [E: XFS_GETQUOTA: Invalid argument] {}
[2018-09-24 14:56:04,479][WARN ][no.found.runner.managers.XFSQuotaWatcher] Command [[sudo, -n, xfs_quota, -x, -c, report -p -n -N]] returned status code [0]] with output: [E: XFS_GETQUOTA: Invalid argument] {}
[2018-09-24 14:56:34,500][WARN ][no.found.runner.managers.XFSQuotaWatcher] Command [[sudo, -n, xfs_quota, -x, -c, report -p -n -N]] returned status code [0]] with output: [E: XFS_GETQUOTA: Invalid argument] {}
6

For me, docker ps:

CONTAINER ID        IMAGE                                                               COMMAND             CREATED             STATUS              PORTS                                                                                                  NAMES
c83a29deb1dd        docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.1.5   "/sbin/my_init"     16 hours ago        Up 16 hours                                                                                                                frc-curators-curator
1cc8677c0ff9        docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.1.5   "/sbin/my_init"     16 hours ago        Up 16 hours         0.0.0.0:12300->12300/tcp, 0.0.0.0:12343->12343/tcp                                                     frc-admin-consoles-admin-console
39fba45a6051        docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.1.5   "/sbin/my_init"     16 hours ago        Up 16 hours         0.0.0.0:12400->5601/tcp, 0.0.0.0:12443->5643/tcp                                                       frc-cloud-uis-cloud-ui
e82cd97b0aef        docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.1.5   "/sbin/my_init"     16 hours ago        Up 16 hours                                                                                                                frc-constructors-constructor
921e2022cc90        docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.1.5   "/sbin/my_init"     16 hours ago        Up 16 hours         0.0.0.0:9244->9244/tcp, 0.0.0.0:12344->12344/tcp                                                       frc-services-forwarders-services-forwarder
e45f71a9f6f4        docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.1.5   "/sbin/my_init"     16 hours ago        Up 16 hours         0.0.0.0:2112->2112/tcp                                                                                 frc-directors-director
d4e9e5ff026e        docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.1.5   "/sbin/my_init"     16 hours ago        Up 16 hours                                                                                                                frc-beats-runners-beats-runner
4f7c7a901cc7        docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.1.5   "/sbin/my_init"     16 hours ago        Up 16 hours                                                                                                                frc-allocators-allocator
0e9190661a45        docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.1.5   "/sbin/my_init"     16 hours ago        Up 16 hours         0.0.0.0:9200->9200/tcp, 0.0.0.0:9243->9243/tcp, 0.0.0.0:9300->9300/tcp, 0.0.0.0:9343->9343/tcp         frc-proxies-proxy
3466690a1d00        docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.1.5   "/sbin/my_init"     16 hours ago        Up 16 hours                                                                                                                frc-blueprints-blueprint
51469d3a948c        docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.1.5   "/sbin/my_init"     16 hours ago        Up 16 hours                                                                                                                frc-runners-runner
6f61b5e95e7d        docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.1.5   "/sbin/my_init"     16 hours ago        Up 16 hours                                                                                                                frc-client-forwarders-client-forwarder
a2584b8711fa        docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.1.5   "/sbin/my_init"     16 hours ago        Up 16 hours         0.0.0.0:2191->2191/tcp, 0.0.0.0:12191->12191/tcp, 0.0.0.0:12898->12898/tcp, 0.0.0.0:13898->13898/tcp   frc-zookeeper-servers-zookeeper
7

@sim

Hmm this is really strange .. the line in the bootstrap log Waiting for [apply-elasticsearch-template] only occurs after Creating Admin Console Elasticsearch backend which only returns when the cluster is created (what are the lines immediately preceding the bootstrap log snippet you provided)

The cluster being created means that from docker ps there should be a container called fac-SOME_CUSTER_ID-instance-0000000000 but there aren't any such containers.

Meanwhile the allocator logs reveal that there was previously a container called fac-252e3238734343dbab69f5b3bb27a6e3-instance-0000000001 ... the 0000000001 meaning that it had been created and then modified from the UI or API at some point, and then deleted. (I don't know if 252e3238734343dbab69f5b3bb27a6e3 is the id of the "adminconsole cluster" it's complaining about)

Is it possible that you reinstalled over a working ECE version without first clearing all the old containers and directories?! That might explain what I'm seeing.

8

@Alex_Piggott

I did do a reinstall but removed the previous install according to elastic docs with docker rm -f frc-runners-runner frc-allocators-allocator $(docker ps -a -q); sudo rm -rf /mnt/data/elastic/* && docker ps -a

I have also removed and reinstalled docker and always with the same results.

I did reach out to elastic support and they were able to reproduce this problem on RedHat 7.5. I confirmed that I was able to complete the install on a 7.4 VM. Waiting to hear back about next steps, but worst case I'll have to downgrade all the systems we've provisioned for this environment. .

9

@sim

Ah found it in our support DB - thanks, I'll check out what's going on (and report any generic findings back here for other people)

Alex

10

My initial problem was caused by not sourcing ALL of the elasticsearch and kibana images. I wrongly assumed it would be ok with 6.4 only.

Once i added the other versions the install went through normally.

1 Like
11

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.