ECE Installation failing, Timeout

[phimob]

Hi,

I'm trying to install ECE on CentOs 7 but it's timing out at "Applying Admin Console Elasticsearch index templates {}"
I can't make much sense of the log file it's telling me to look at.
Any help would be appreciated.

I'm running this command:
bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install

This it the output:

-- Verifying Prerequisites --
Checking runner container does not exist... PASSED
Checking host storage root volume path is not root... PASSED
Checking host storage path is accessible... PASSED
Checking host storage path contents matches whitelist... PASSED
Checking Docker version... PASSED
Checking Docker file system... PASSED
Checking Docker storage driver... PASSED
Checking runner ip connectivity... PASSED
Checking OS IPv4 IP forward setting... PASSED
Checking OS max map count setting... PASSED
Checking OS kernel version... PASSED
Checking minimum required memory... PASSED
Checking OS kernel cgroup.memory... PASSED
Checking OS minimum ephemeral port... PASSED
Checking OS max open file descriptors per process... PASSED
Checking OS max open file descriptors system-wide... PASSED
Checking OS file system and Docker storage driver compatibility... PASSED
-- Completed Verifying Prerequisites --

• Running Bootstrap container
• Monitoring bootstrap process
• Loaded bootstrap settings {}
• Initialising feature flag [DedicatedNodeTypes] to [true] {}
• Starting local runner {}
• Started local runner {}
• Waiting for runner container node {}
• Runner container node detected {}
• Waiting for coordinator candidate {}
• Detected coordinator candidate {}
• Detected pending coordinator, promoting coordinator {}
• Coordinator accepted {}
• Storing current platform version: 2.0.0 {}
• Storing Instance Types: [elasticsearch,kibana] {}
• Storing Elastic Stack versions: [5.6.12,6.4.1] {}
• Creating Admin Console Elasticsearch backend {}
• Applying Admin Console Elasticsearch index templates {}
• Unhandled error. {}
  -- An error has occurred in bootstrap process. Please examine logs --
  java.util.concurrent.TimeoutException: Futures timed out after [600000 milliseconds]
  at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:223)
  at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:227)
  at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:190)
  at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
  at scala.concurrent.Await$.result(package.scala:190)
  at no.found.bootstrap.BootstrapInitial.bootstrapServiceLayer(BootstrapInitial.scala:963)
  at no.found.bootstrap.BootstrapInitial.bootstrap(BootstrapInitial.scala:678)
  at no.found.bootstrap.BootstrapInitial$.delayedEndpoint$no$found$bootstrap$BootstrapInitial$1(BootstrapInitial.scala:1311)
  at no.found.bootstrap.BootstrapInitial$delayedInit$body.apply(BootstrapInitial.scala:1305)
  at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
  at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
  at scala.App$$anonfun$main$1.apply(App.scala:76)
  at scala.App$$anonfun$main$1.apply(App.scala:76)
  at scala.collection.immutable.List.foreach(List.scala:392)
  at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
  at scala.App$class.main(App.scala:76)
  at no.found.bootstrap.BootstrapInitial$.main(BootstrapInitial.scala:1305)
  at no.found.bootstrap.BootstrapInitial.main(BootstrapInitial.scala)

  Errors have caused Elastic Cloud Enterprise installation to fail - Please check logs
  Node type - initial

[phimob]

This is the last couple of lines of the log file:

[2018-11-23 13:10:45,662][INFO ][no.found.bootstrap.BootstrapInitial] Creating Admin Console Elasticsearch backend {}
[2018-11-23 13:10:48,184][INFO ][no.found.bootstrap.ServiceLayerBootstrap] Waiting for [ensuring-plan] to complete. Retrying every [1 second] (cause: [org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /clusters/cb1e4ef66659415c8f2320bc3256170d/plans/status]) {}
[2018-11-23 13:11:33,214][INFO ][no.found.bootstrap.ServiceLayerBootstrap] Waiting for [ensuring-plan] to complete. Retrying every [1 second] (cause: [java.lang.Exception: not yet started]) {}
[2018-11-23 13:12:09,214][INFO ][no.found.bootstrap.ServiceLayerBootstrap] Waiting for [ensuring-plan] to complete. Retrying every [1 second] (cause: [org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /clusters/cb1e4ef66659415c8f2320bc3256170d/plans/pending]) {}
[2018-11-23 13:12:16,442][INFO ][no.found.bootstrap.BootstrapInitial] Applying Admin Console Elasticsearch index templates {}
[2018-11-23 13:12:16,567][INFO ][no.found.bootstrap.ServiceLayerBootstrap] Waiting for [apply-elasticsearch-template] to complete. Retrying every [1 second] (cause: [java.lang.Exception: 410 Gone: HttpEntity(application/json; charset=UTF-8,{"ok":false,"message":"Deleted cluster."})]) {}
[2018-11-23 13:12:17,613][INFO ][no.found.bootstrap.ServiceLayerBootstrap] Waiting for [apply-elasticsearch-template] to complete. Retrying every [1 second] (cause: [java.lang.Exception: 502 Bad Gateway: HttpEntity(application/json; charset=UTF-8,{"ok":false,"message":"The requested cluster is currently unavailable"})]) {}
[2018-11-23 13:12:43,354][INFO ][no.found.bootstrap.ServiceLayerBootstrap] Waiting for [apply-elasticsearch-template] to complete. Retrying every [1 second] (cause: [java.lang.Exception: 410 Gone: HttpEntity(application/json; charset=UTF-8,{"ok":false,"message":"Deleted cluster."})]) {}
[2018-11-23 13:22:16,460][ERROR][no.found.bootstrap.BootstrapInitial$] Unhandled error. {}
java.util.concurrent.TimeoutException: Futures timed out after [600000 milliseconds]
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:223)
at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:227)
at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:190)
at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
at scala.concurrent.Await$.result(package.scala:190)
at no.found.bootstrap.BootstrapInitial.bootstrapServiceLayer(BootstrapInitial.scala:963)
at no.found.bootstrap.BootstrapInitial.bootstrap(BootstrapInitial.scala:678)
at no.found.bootstrap.BootstrapInitial$.delayedEndpoint$no$found$bootstrap$BootstrapInitial$1(BootstrapInitial.scala:1311)
at no.found.bootstrap.BootstrapInitial$delayedInit$body.apply(BootstrapInitial.scala:1305)
at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
at scala.App$$anonfun$main$1.apply(App.scala:76)
at scala.App$$anonfun$main$1.apply(App.scala:76)
at scala.collection.immutable.List.foreach(List.scala:392)
at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
at scala.App$class.main(App.scala:76)
at no.found.bootstrap.BootstrapInitial$.main(BootstrapInitial.scala:1305)
at no.found.bootstrap.BootstrapInitial.main(BootstrapInitial.scala)
[2018-11-23 13:22:16,510][INFO ][no.found.bootstrap.ServiceLayerBootstrap] Waiting for [apply-elasticsearch-template] to complete. Retrying every [1 second] (cause: [akka.pattern.AskTimeoutException: Recipient[Actor[akka://default/user/IO-HTTP#-833220481]] had already been terminated. Sender[null] sent the message of type "spray.http.HttpRequest".]) {}

Thanks for your time.

[Alex_Piggott]

Thanks for the report, that's a new one for me! We're looking into this now (we haven't had widespread reports on this, so I'm guessing "shooting from the hip" that it's an OS version incompatibility problem)

[Alex_Piggott]

What Kernel version are you running? There does appear to be some reports of incompatibility vs some Centos 7 kernels

[phimob]

Hi, i'm running 4.4.162-1.el7.elrepo.x86_64

[Alex_Piggott]

I think this applies (Limitations and known problems | Elastic Cloud Enterprise Reference [3.6] | Elastic):

• When you use OverlayFS with Kernel-LT 4.4.156 and later, there is a known regression that prevents Elastic Cloud Enterprise from completing the installation. From the kernel-archive, use Kernel 4.4.155.

See Install Failing ECE 2.0.0 CentOS 7.5 - #5 by naknak987 for more details

[phimob]

That was the problem, it's working now. Thank you very much!