ECE failed to install

Hi,

I'm running into an issue when running the offline installation of ECE 1.1.1 on RHEL7
After running through the prerequisites, here's the output of the install with the --debug switch
(sorry had to split the logs over several posts due to character limit)

        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    Elastic Cloud Enterprise Installer

    Start setting up a new Elastic Cloud Enterprise installation by installing the software on your first host.
    This first host becomes the initial coordinator and provides access to the Cloud UI, where you can manage your installation.
    To learn more about the options you can specify, see the documentation.

    NOTE: If you want to add this host to an existing installation, please specify the --coordinator-host and --roles-token flags
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

    -- Debug Info --
    NODE_TYPE - initial
    CLOUD_ENTERPRISE_VERSION - 1.1.1
    HOST_DOCKER_HOST - /var/run/docker.sock
    HOST_STORAGE_DEVICE_PATH - /dev/mapper/gcs1vg-datalv
    HOST_STORAGE_ROOT_VOLUME_PATH - /export/gcs1/data
    HOST_STORAGE_PATH - /export/gcs1/data/elastic
    NODE_TYPE - initial
    RUNNER_REGION - ece-region
    RUNNER_ID - 10.244.49.55
    RUNNER_ROLES -
    RUNNER_ROLES_TOKEN -
    RUNNER_HOST_IP - 10.244.49.55
    RUNNER_AVAILABILITY_ZONE - ece-region-1a
    RUNNER_CAPACITY - 27211
    RUNNER_ENVIRONMENT_METADATA_JSON - {}
    ELASTIC_UID - 1000048982
    ELASTIC_GID - 1000008459
    ELASTIC_CLOUD_PLATFORM - aws
    MEMORY_SETTINGS - {}
    CONFIG_FILE -
    FORCE_INSTALL - False
    HOST_PREREQ_FAILED - False
    DOCKER_REGISTRY - docker.elastic.co
    ECE_DOCKER_REPOSITORY - cloud-enterprise
    ECE_TIMEOUT_FACTOR - 1.0

    -- Verifying Prerequisites --
    Checking host storage path... PASSED
    Checking docker version... PASSED
    Checking internal ip connectivity... PASSED
    Checking OS settings... PASSED
    Checking OS kernel version... PASSED
     - OS kernel version is 3.10.0-514.26.2.el7.x86_64 but we recommend 4.4.
    Checking Minimum required memory... PASSED
    Checking Kernel cgroup.memory setting... PASSED
     - OS setting 'cgroup.memory' should be set to cgroup.memory=nokmem
    -- Completed Verifying Prerequisites --

    - Running Bootstrap container
    - Monitoring bootstrap process
    - [2017-12-05 12:56:20,898][INFO ][no.found.bootstrap.BootstrapInitial] Loaded bootstrap settings {}
    - [2017-12-05 12:56:27,358][WARN ][no.found.docker.DockerContainerManager] Default registry [docker.elastic.co] has no auths. Known auths: [List()] {}
    - [2017-12-05 12:56:28,996][INFO ][no.found.docker.DockerContainerManager] Starting container [frc-zookeeper-servers-zookeeper] {"ec_container_kind":"docker","ec_container_group":"zookeeper-servers","ec_container_name":"zookeeper"}
    - [2017-12-05 12:56:28,996][INFO ][no.found.docker.DockerContainerManager] Starting container [frc-client-forwarders-client-forwarder] {"ec_container_kind":"docker","ec_container_group":"client-forwarders","ec_container_name":"client-forwarder"}
    - [2017-12-05 12:56:29,324][INFO ][org.apache.curator.framework.imps.CuratorFrameworkImpl] Starting {}
    - [2017-12-05 12:56:29,343][INFO ][no.found.curator.ForwardedEnsembleProvider] Unable to read servers list from [http://10.244.49.55:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1], falling back to [0.0.0.0:2181] {}
    - [2017-12-05 12:56:29,344][INFO ][no.found.curator.ForwardedEnsembleProvider] Resolved connection string from [http://10.244.49.55:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1] to [0.0.0.0:2181/v1] with local namespace [/v1] {}
    - [2017-12-05 12:56:29,366][INFO ][no.found.curator.ForwardedEnsembleProvider] Unable to read servers list from [http://10.244.49.55:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1], falling back to [0.0.0.0:2181] {}
    - [2017-12-05 12:56:29,367][INFO ][no.found.curator.ForwardedEnsembleProvider] Resolved connection string from [http://10.244.49.55:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1] to [0.0.0.0:2181/v1] with local namespace [/v1] {}
    - [2017-12-05 12:56:29,369][INFO ][org.apache.curator.framework.imps.CuratorFrameworkImpl]
    Curator Schemas:
    -
    -  {}
    - [2017-12-05 12:56:30,376][INFO ][no.found.curator.ForwardedEnsembleProvider] Unable to read servers list from [http://10.244.49.55:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1], falling back to [0.0.0.0:2181] {}
    - [2017-12-05 12:56:30,377][INFO ][no.found.curator.ForwardedEnsembleProvider] Resolved connection string from [http://10.244.49.55:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1] to [0.0.0.0:2181/v1] with local namespace [/v1] {}
    - [2017-12-05 12:56:31,378][INFO ][no.found.curator.ForwardedEnsembleProvider] Unable to read servers list from [http://10.244.49.55:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1], falling back to [0.0.0.0:2181] {}
    - [2017-12-05 12:56:31,379][INFO ][no.found.curator.ForwardedEnsembleProvider] Resolved connection string from [http://10.244.49.55:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1] to [0.0.0.0:2181/v1] with local namespace [/v1] {}
    - [2017-12-05 12:56:32,380][INFO ][no.found.curator.ForwardedEnsembleProvider] Unable to read servers list from [http://10.244.49.55:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1], falling back to [0.0.0.0:2181] {}
    - [2017-12-05 12:56:32,380][INFO ][no.found.curator.ForwardedEnsembleProvider] Resolved connection string from [http://10.244.49.55:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1] to [0.0.0.0:2181/v1] with local namespace [/v1] {}
    - [2017-12-05 12:56:33,382][INFO ][no.found.curator.ForwardedEnsembleProvider] Unable to read servers list from [http://10.244.49.55:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1], falling back to [0.0.0.0:2181] {}
    - [2017-12-05 12:56:33,382][INFO ][no.found.curator.ForwardedEnsembleProvider] Resolved connection string from [http://10.244.49.55:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1] to [0.0.0.0:2181/v1] with local namespace [/v1] {}
    - [2017-12-05 12:56:34,586][INFO ][no.found.curator.ForwardedEnsembleProvider] Resolved connection string from [http://10.244.49.55:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1] to [10.244.49.55:22191/v1] with local namespace [/v1] {}
    - [2017-12-05 12:56:34,586][INFO ][org.apache.curator.ConnectionState] Connection string changed to: 0.0.0.0:2181/v1 {}
    - [2017-12-05 12:56:34,979][INFO ][no.found.curator.ForwardedEnsembleProvider] Resolved connection string from [http://10.244.49.55:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1] to [10.244.49.55:22191/v1] with local namespace [/v1] {}
    - [2017-12-05 12:56:35,011][INFO ][no.found.curator.ForwardedEnsembleProvider] Resolved connection string from [http://10.244.49.55:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1] to [10.244.49.55:22191/v1] with local namespace [/v1] {}
    - [2017-12-05 12:56:35,015][INFO ][org.apache.curator.framework.state.ConnectionStateManager] State change: CONNECTED {}
    - [2017-12-05 12:56:35,035][INFO ][org.apache.curator.framework.imps.EnsembleTracker] New config event received: {version=100000000, server.10=localhost:2898:3898:participant;0.0.0.0:2191} {}
    - [2017-12-05 12:56:35,066][INFO ][org.apache.curator.framework.imps.CuratorFrameworkImpl] Starting {}
    - [2017-12-05 12:56:35,069][INFO ][no.found.curator.ForwardedEnsembleProvider] Resolved connection string from [http://10.244.49.55:2180/zookeeper/clients/ensemble/connection-string?namespace=/] to [10.244.49.55:22191/] with local namespace [/] {}
    - [2017-12-05 12:56:35,074][INFO ][no.found.curator.ForwardedEnsembleProvider] Resolved connection string from [http://10.244.49.55:2180/zookeeper/clients/ensemble/connection-string?namespace=/] to [10.244.49.55:22191/] with local namespace [/] {}
    - [2017-12-05 12:56:35,074][INFO ][org.apache.curator.framework.imps.CuratorFrameworkImpl] Curator Schemas:
    -
    -  {}
    - [2017-12-05 12:56:35,078][INFO ][no.found.curator.ForwardedEnsembleProvider] Resolved connection string from [http://10.244.49.55:2180/zookeeper/clients/ensemble/connection-string?namespace=/] to [10.244.49.55:22191/] with local namespace [/] {}
    - [2017-12-05 12:56:35,079][INFO ][org.apache.curator.framework.state.ConnectionStateManager]
    State change: CONNECTED {}
    - [2017-12-05 12:56:35,083][INFO ][org.apache.curator.framework.imps.EnsembleTracker] New config event received: {version=100000000, server.10=localhost:2898:3898:participant;0.0.0.0:2191} {}
    - [2017-12-05 12:56:35,095][INFO ][org.apache.curator.framework.imps.CuratorFrameworkImpl] backgroundOperationsLoop exiting {}
    - [2017-12-05 12:56:35,335][INFO ][no.found.bootstrap.BootstrapInitial] Creating container set [runners] with acls: List(31,s{'digest,'runners:c7iikXun0mpNk3UQYYPUA6XNEHQ=}
    - , 31,s{'digest,'runners:Uaa2vSluWAadbFoS2bvbp+r9VP0=}
    - ) {}
    - [2017-12-05 12:56:35,369][INFO ][no.found.bootstrap.BootstrapInitial] Creating container set [zookeeper-servers] with acls: List(31,s{'digest,'runners:A/pNRPZSZtzXZsXFqJNhwuIFBqo=}
    - , 31,s{'digest,'zookeeper-servers:TTyddzOJHvLezmvNZO5sZatTrHM=}
    - ) {}
    - [2017-12-05 12:56:35,382][INFO ][no.found.bootstrap.BootstrapInitial] Creating container set [client-forwarders] with acls: List(31,s{'digest,'runners:mE7cMTGJWNzNDG4BZFLlmiMVD+s=}
    - , 31,s{'digest,'client-forwarders:IQ+DE7SQboKbF8pBHZBdIu66scs=}
    - ) {}
    - [2017-12-05 12:56:35,395][INFO ][no.found.bootstrap.BootstrapInitial] Creating container set [client-observers] with acls: List(31,s{'digest,'runners:WWOS2xn/ckcPYmKalIWyynmwYdo=}
    - , 31,s{'digest,'client-observers:M6J6Nl/PZ6uJ8uOXn78AZUF+2j8=}
    - ) {}
    - [2017-12-05 12:56:35,408][INFO ][no.found.bootstrap.BootstrapInitial] Creating container set [directors] with acls: List(31,s{'digest,'runners:I32CqYn2mVFkMDaGSM98mmko7/A=}
    - , 31,s{'digest,'directors:GZi4ZGhNcer9bksNjUIWvlPsSiE=}
    - ) {}
    - [2017-12-05 12:56:35,417][INFO ][no.found.bootstrap.BootstrapInitial] Creating container set [blueprints] with acls: List(31,s{'digest,'runners:hgWIM8Czo3L0jOVddBXrANQPfiI=}
    - , 31,s{'digest,'blueprints:NcEBEtA9JEspUNsiK4HhIwLPgAo=}
    - ) {}
    - [2017-12-05 12:56:35,453][INFO ][no.found.bootstrap.BootstrapInitial] Creating container set [services-forwarders] with acls: List(31,s{'digest,'runners:ZtaxOBNx8Ww7CXcFyTbv5MKZQjk=}
    - , 31,s{'digest,'services-forwarders:MIfCtkyPTwnoieG5riNTLQMFxj4=}
    - ) {}
    - [2017-12-05 12:56:35,466][INFO ][no.found.bootstrap.BootstrapInitial] Creating container set [constructors] with acls: List(31,s{'digest,'runners:Y2Aq/sWcwKJ2GdAVeNWlXhhEn28=}
    - , 31,s{'digest,'constructors:yWCrsExTtXeUkep1xvp6jES0jiI=}
    - ) {}
    - [2017-12-05 12:56:35,479][INFO ][no.found.bootstrap.BootstrapInitial] Creating container set [allocators] with acls: List(31,s{'digest,'runners:7bOmBev1WHwDvk+Wzh2rEdBIXJU=}
    - , 31,s{'digest,'allocators:EQJC2MAL/Amt/Y1R09rkqiQY+eg=}
    - ) {}
    - [2017-12-05 12:56:35,494][INFO ][no.found.bootstrap.BootstrapInitial] Creating container set [admin-consoles] with acls: List(31,s{'digest,'runners:LrR6Lid7zwuD6AIhcFSjlW4WSJM=}
    - , 31,s{'digest,'admin-consoles:m0UZixLESuLpeb0vB1soqwlV/dM=}
    - ) {}
    - [2017-12-05 12:56:35,792][INFO ][no.found.bootstrap.BootstrapInitial] Creating container set [cloud-uis] with acls: List(31,s{'digest,'runners:FxHlBVbCyNqfMJczyxh49XbIt3A=}
    - , 31,s{'digest,'cloud-uis:KxTaZIyqtG1GnQRyl5DX6bJDntQ=}
    - ) {}
    - [2017-12-05 12:56:35,802][INFO ][no.found.bootstrap.BootstrapInitial] Creating container set [proxies] with acls: List(31,s{'digest,'runners:bJwfCpHg7bZyKCAu7GEXtjI6kjE=}
    - , 31,s{'digest,'proxies:Ag2IWO1uGTaLN8kP5OByT60T+1k=}
    - ) {}
    - [2017-12-05 12:56:35,818][INFO ][no.found.bootstrap.BootstrapInitial] Creating container set [beats-runners] with acls: List(31,s{'digest,'runners:cif5dqejeNgg9lPc850iF3x07Mc=}
    - , 31,s{'digest,'beats-runners:ILF1UoS5AmY4PGUuB9ylkFurCeU=}
    - ) {}
    - [2017-12-05 12:56:35,835][INFO ][no.found.bootstrap.BootstrapInitial] Creating container set [curators] with acls: List(31,s{'digest,'runners:uIXr6it/XRoM0TJZ7/zVw37zDcE=}
    - , 31,s{'digest,'curators:Adr3QMLjWt766ZSM5JLrcV65tA8=}
    - ) {}
    - [2017-12-05 12:56:35,849][INFO ][no.found.bootstrap.BootstrapInitial] Starting local runner {}
    - [2017-12-05 12:56:35,866][INFO ][no.found.docker.DockerContainerManager] Starting container [frc-runners-runner] {"ec_container_kind":"docker","ec_container_group":"runners","ec_container_name":"runner"}
    - [2017-12-05 12:56:35,964][INFO ][no.found.bootstrap.BootstrapInitial] Started local runner {}
    - [2017-12-05 12:56:35,966][INFO ][no.found.bootstrap.BootstrapInitial] Waiting for runner container node {}
    - [2017-12-05 12:56:35,982][INFO ][org.apache.curator.framework.imps.EnsembleTracker] New config event received: {version=100000000, server.10=localhost:2898:3898:participant;0.0.0.0:2191} {}
    - [2017-12-05 12:56:39,555][INFO ][no.found.bootstrap.BootstrapInitial] Runner container node detected {}
    - [2017-12-05 12:56:39,733][INFO ][no.found.bootstrap.BootstrapInitial] Waiting for coordinator candidate {}
    - [2017-12-05 12:56:52,268][INFO ][no.found.bootstrap.BootstrapInitial] Detected coordinator candidate {}
    - [2017-12-05 12:56:52,311][INFO ][no.found.bootstrap.BootstrapInitial] Detected pending coordinator, promoting coordinator {}
    - [2017-12-05 12:56:52,422][INFO ][no.found.zookeeper.models.coordinators.CoordinatorCandidatePendingInfo] Current instances: [4294967329,4294967329,1512478595214,1512478595214,0,0,0,0,2,0,4294967329], secrets: [4294967331,4294967331,1512478595224,1512478595224,0,0,0,0,20062,0,4294967331] {}
    - [2017-12-05 12:56:52,467][INFO ][no.found.bootstrap.BootstrapInitial] Coordinator accepted {}
    - [2017-12-05 12:56:52,468][INFO ][no.found.bootstrap.BootstrapInitial] Storing current platform version: 1.1.1 {}
    - [2017-12-05 12:56:52,519][INFO ][no.found.bootstrap.BootstrapInitial] Storing Elastic Stack versions: [2.4.6,5.6.4,6.0.0] {}
    - [2017-12-05 12:56:52,887][INFO ][no.found.bootstrap.BootstrapInitial] Creating Admin Console Elasticsearch backend {}
    - [2017-12-05 12:56:53,247][INFO ][no.found.bootstrap.ServiceLayerBootstrap] Waiting for [ensuring-plan] to complete. Retrying every [1 second] (cause: [org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /clusters/4f159ef668d941c9a727c7d825f013ad/plans/status]) {}
    - [2017-12-05 12:57:03,502][INFO ][no.found.bootstrap.ServiceLayerBootstrap] Waiting for [ensuring-plan] to complete. Retrying every [1 second] (cause: [java.lang.Exception: not yet started]) {}
    - [2017-12-05 12:57:34,170][INFO ][no.found.bootstrap.BootstrapInitial] Applying Admin Console Elasticsearch index templates {}
    - [2017-12-05 12:57:34,182][INFO ][no.found.bootstrap.ServiceLayerBootstrap] Waiting for [apply-elasticsearch-template] to complete. Retrying every [1 second] (cause: [spray.can.Http$ConnectionAttemptFailedException: Connection attempt to containerhost:9244 failed]) {}
[2017-12-05 13:07:34,174][ERROR][no.found.bootstrap.BootstrapInitial$] Unhandled error. {}
-- An error has occurred in bootstrap process. Please examine logs --
- java.util.concurrent.TimeoutException: Futures timed out after [600000 milliseconds]
- 	at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
- 	at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
- 	at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:190)
- 	at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
- 	at scala.concurrent.Await$.result(package.scala:190)
- 	at no.found.bootstrap.BootstrapInitial.bootstrapServiceLayer(BootstrapInitial.scala:884)
- 	at no.found.bootstrap.BootstrapInitial.bootstrap(BootstrapInitial.scala:642)
- 	at no.found.bootstrap.BootstrapInitial$.delayedEndpoint$no$found$bootstrap$BootstrapInitial$1(BootstrapInitial.scala:1207)
- 	at no.found.bootstrap.BootstrapInitial$delayedInit$body.apply(BootstrapInitial.scala:1201)
- 	at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
- 	at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
- 	at scala.App$$anonfun$main$1.apply(App.scala:76)
- 	at scala.App$$anonfun$main$1.apply(App.scala:76)
- 	at scala.collection.immutable.List.foreach(List.scala:381)
- 	at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
- 	at scala.App$class.main(App.scala:76)
- 	at no.found.bootstrap.BootstrapInitial$.main(BootstrapInitial.scala:1201)
- 	at no.found.bootstrap.BootstrapInitial.main(BootstrapInitial.scala)
- [2017-12-05 13:07:34,200][INFO ][no.found.bootstrap.ServiceLayerBootstrap] Waiting for [apply-elasticsearch-template] to complete. Retrying every [1 second] (cause: [akka.pattern.AskTimeoutException: Recipient[Actor[akka://default/user/IO-HTTP#1219371542]] had already been terminated.]) {}
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  Errors have caused Elastic Cloud Enterprise installation to fail - Please check logs
  Node type - initial
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Any ideas on what is going wrong?

Thanks,

Aaron

And here's the output of docker ps -a

CONTAINER ID        IMAGE                                                               COMMAND                  CREATED             STATUS                     PORTS                                                                                                  NAMES
a45f8226e1c6        docker.elastic.co/cloud-assets/elasticsearch:5.6.4-0                "/sbin/entry-point"      14 minutes ago      Up 14 minutes              0.0.0.0:18478->18478/tcp, 0.0.0.0:19780->19780/tcp                                                     fac-4f159ef668d941c9a727c7d825f013ad-instance-0000000000
e5c6eeb3118a        docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.1.1   "/sbin/my_init"          14 minutes ago      Up 14 minutes                                                                                                                     frc-curators-curator
55bdafbc7cf5        docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.1.1   "/sbin/my_init"          14 minutes ago      Up 14 minutes              0.0.0.0:12400->5601/tcp, 0.0.0.0:12443->5643/tcp                                                       frc-cloud-uis-cloud-ui
256be28e42fb        docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.1.1   "/sbin/my_init"          14 minutes ago      Up 14 minutes              0.0.0.0:12300->12300/tcp, 0.0.0.0:12343->12343/tcp                                                     frc-admin-consoles-admin-console
646436124d4d        docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.1.1   "/sbin/my_init"          14 minutes ago      Up 14 minutes                                                                                                                     frc-constructors-constructor
8b116fd3d255        docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.1.1   "/sbin/my_init"          15 minutes ago      Up 15 minutes              0.0.0.0:2112->2112/tcp                                                                                 frc-directors-director
83faf6ebf717        docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.1.1   "/sbin/my_init"          15 minutes ago      Up 15 minutes                                                                                                                     frc-beats-runners-beats-runner
24fdf2728b71        docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.1.1   "/sbin/my_init"          15 minutes ago      Up 15 minutes                                                                                                                     frc-allocators-allocator
20002ba133a7        docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.1.1   "/sbin/my_init"          15 minutes ago      Up 15 minutes              0.0.0.0:9200->9200/tcp, 0.0.0.0:9243->9243/tcp, 0.0.0.0:9300->9300/tcp, 0.0.0.0:9343->9343/tcp         frc-proxies-proxy
9dc176e328b1        docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.1.1   "/sbin/my_init"          15 minutes ago      Up 15 minutes                                                                                                                     frc-blueprints-blueprint
890d682149d6        docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.1.1   "/sbin/my_init"          15 minutes ago      Up 15 minutes                                                                                                                     frc-runners-runner
d547434972d0        docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.1.1   "/sbin/my_init"          15 minutes ago      Up 15 minutes              0.0.0.0:2191->2191/tcp, 0.0.0.0:12191->12191/tcp, 0.0.0.0:12898->12898/tcp, 0.0.0.0:13898->13898/tcp   frc-zookeeper-servers-zookeeper
094e8704fc9b        docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.1.1   "/sbin/my_init"          15 minutes ago      Up 15 minutes                                                                                                                     frc-client-forwarders-client-forwarder
40592e28a002        docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.1.1   "/sbin/my_init"          15 minutes ago      Exited (0) 4 minutes ago                                                                                                          elastic-cloud-enterprise-bootstrap-1.1.1
e9aab26a1813        docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.1.1   "elastic-cloud-enterp"   15 minutes ago      Dead                                                                                                                              elastic-cloud-enterprise-installer-1.1.1

@adesouza

To close the loop on this, was this issue fixed by clearing out some docker stuff before re-installing, eg

    docker rm -f $(docker ps -a -q)
    docker volume rm $(docker volume ls -q)

(followed by reboot and reinstall?)

Oh, yes. My first install failed because for another reason (wrong images loaded into docker), and when I tried to reinstall one of the container names was already registered in docker which stopped it from starting another one with the same name. Clearing out all of the containers and volumes and then performing the install again worked.

Thanks,

Aaron

I'm having exactly the same issue. I've removed a previous installation using the steps above, but I'm still getting the same error when I attempt to re-install. I must be missing a step, but can't figure out what.

@crickes

Can you confirm the following, to double check that it is the same issue ... if so, you should have an error like the following in your runner.log (eg /mnt/data/elastic/<<RUNNERID>>/services/runner/logs/runner.log):

[2017-MM-DD hh:mm:ss.SSS][INFO ][no.found.docker.DockerContainerManager] Container [frc-services-forwarders-services-forwarder] exists, could not create: [HttpResponse(409 Conflict,HttpEntity(text/plain; charset=UTF-8,Conflict. The name "/frc-services-forwarders-services-forwarder" is already in use by container <<CONTAINERID>>. You have to remove (or rename) that container to be able to reuse that name.

If so, a reboot after removing the docker volumes should fix it. You could uninstall docker, delete the docker dir eg /mnt/data/docker and try reinstalling?

I suspect its the /mnt/data/docker/overlay directory. There are lots of directories in there, some created by the elastic user, and some by root. I remember seeing an error in a log file about not being able to delete something after installation.

I've just deleted the contents of that manually, and will try reboot / reinstall now.

Yep, had to remove and recreate the /mnt/data/docker directory and reboot, before it would re-install correctly.

Even though I change the /mnt/data/ to be owned by the elastic user and group, when it installs, it still creates paths as the root user and then isn't able to clean up the installation afterwards. So I get this message at the end of the installation:

Error response from daemon: Unable to remove filesystem for cce0cdb9432bbc4d107fbbaf11eb6810bbe75ca2c191a9669eb128a39c549b72: remove /mnt/data/docker/containers/cce0cdb9432bbc4d107fbbaf11eb6810bbe75ca2c191a9669eb128a39c549b72/shm: device or resource busy

Everything in /mnt/data/docker is now owned by root again.

@crickes

Glad you got it working! Thanks for sharing the steps

btw I think Error response from daemon: Unable to remove filesystem for cce0cdb9432bbc4d107fbbaf11eb6810bbe75ca2c191a9669eb128a39c549b72: remove /mnt/data/docker/containers/cce0cdb9432bbc4d107fbbaf11eb6810bbe75ca2c191a9669eb128a39c549b72/shm: device or resource busy is a separate issue (which is mostly harmless in our case)

See this docker issue for more details

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.