Install Failing ECE 2.0.0 CentOS 7.5

Elastic Orchestration Elastic Cloud Enterprise (ECE)
1

I have a CentOS 7.5 VM that I'm trying to install ECE 2.0.0 on. I've setup everything according to https://www.elastic.co/guide/en/cloud-enterprise/current/ece-configure-hosts.html#ece-configure-hosts-rhel-centos.

I can't find anything on here or google that seems to help. Below is the output of bash <(curl -fsSL https://download.elastic.co/cloud/elastic-cloud-enterprise.sh) install --debug.

See the following post, it was too long to put here.

After that, I get the following from running docker ps.

CONTAINER ID        IMAGE                                                               COMMAND             CREATED             STATUS              PORTS                                                                                                  NAMES
23362ca36f11        docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:2.0.0   "/sbin/my_init"     34 minutes ago      Up 34 minutes       0.0.0.0:12400->5601/tcp, 0.0.0.0:12443->5643/tcp                                                       frc-cloud-uis-cloud-ui
0a413c449a4e        docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:2.0.0   "/sbin/my_init"     34 minutes ago      Up 34 minutes       0.0.0.0:12300->12300/tcp, 0.0.0.0:12343->12343/tcp                                                     frc-admin-consoles-admin-console
7efcfd93df5f        docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:2.0.0   "/sbin/my_init"     34 minutes ago      Up 34 minutes                                                                                                              frc-constructors-constructor
de4b8d73d88a        docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:2.0.0   "/sbin/my_init"     34 minutes ago      Up 34 minutes                                                                                                              frc-curators-curator
da684012f685        docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:2.0.0   "/sbin/my_init"     34 minutes ago      Up 34 minutes       0.0.0.0:9244->9244/tcp, 0.0.0.0:12344->12344/tcp                                                       frc-services-forwarders-services-forwarder
dc730f25a56e        docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:2.0.0   "/sbin/my_init"     34 minutes ago      Up 34 minutes       0.0.0.0:2112->2112/tcp                                                                                 frc-directors-director
5fc99876ee5c        docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:2.0.0   "/sbin/my_init"     34 minutes ago      Up 34 minutes                                                                                                              frc-beats-runners-beats-runner
00f1c387ab3d        docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:2.0.0   "/sbin/my_init"     34 minutes ago      Up 34 minutes                                                                                                              frc-allocators-allocator
4e72934ddb53        docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:2.0.0   "/sbin/my_init"     34 minutes ago      Up 34 minutes       0.0.0.0:9200->9200/tcp, 0.0.0.0:9243->9243/tcp, 0.0.0.0:9300->9300/tcp, 0.0.0.0:9343->9343/tcp         frc-proxies-proxy
3018942d9ad0        docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:2.0.0   "/sbin/my_init"     34 minutes ago      Up 34 minutes                                                                                                              frc-blueprints-blueprint
0225dbed61c0        docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:2.0.0   "/sbin/my_init"     34 minutes ago      Up 34 minutes                                                                                                              frc-runners-runner
8564896f7720        docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:2.0.0   "/sbin/my_init"     34 minutes ago      Up 34 minutes                                                                                                              frc-client-forwarders-client-forwarder
10f97d0dbf31        docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:2.0.0   "/sbin/my_init"     34 minutes ago      Up 34 minutes       0.0.0.0:2191->2191/tcp, 0.0.0.0:12191->12191/tcp, 0.0.0.0:12898->12898/tcp, 0.0.0.0:13898->13898/tcp   frc-zookeeper-servers-zookeeper
2 
-- Debug Info --
NODE_TYPE - initial
CLOUD_ENTERPRISE_VERSION - 2.0.0
HOST_DOCKER_HOST - /var/run/docker.sock
HOST_STORAGE_DEVICE_PATH - /dev/mapper/centos-root
HOST_STORAGE_ROOT_VOLUME_PATH - /mnt/data
HOST_STORAGE_PATH - /mnt/data/elastic
NODE_TYPE - initial
RUNNER_REGION - ece-region
RUNNER_ID - 172.16.10.130
RUNNER_ROLES -
RUNNER_ROLES_TOKEN -
RUNNER_HOST_IP - 172.16.10.130
RUNNER_AVAILABILITY_ZONE - ece-zone-1
RUNNER_CAPACITY - 27342
RUNNER_ENVIRONMENT_METADATA_JSON - {}
ALLOCATOR_TAGS -
ELASTIC_UID - 1000
ELASTIC_GID - 1001
ELASTIC_CLOUD_PLATFORM - aws
MEMORY_SETTINGS - {}
CONFIG_FILE -
FORCE_INSTALL - False
HOST_PREREQ_FAILED - False
DOCKER_REGISTRY - docker.elastic.co
ECE_DOCKER_REPOSITORY - cloud-enterprise
HOST_KERNEL_PARAMETERS - {'net.ipv4.ip_local_port_range': '32768\t60999'}
CLOUD_FEATURE_FLAG_DEDICATEDNODETYPES - true
ECE_TIMEOUT_FACTOR - 1.0

-- Verifying Prerequisites --
Checking runner container does not exist... PASSED
Checking host storage root volume path is not root... PASSED
Checking host storage path is accessible... PASSED
Checking host storage path contents matches whitelist... PASSED
Checking Docker version... PASSED
Checking Docker file system... PASSED
Checking Docker storage driver... PASSED
Checking runner ip connectivity... PASSED
Checking OS IPv4 IP forward setting... PASSED
Checking OS max map count setting... PASSED
Checking OS kernel version... PASSED
Checking minimum required memory... PASSED
Checking OS kernel cgroup.memory... PASSED
Checking OS minimum ephemeral port... PASSED
Checking OS max open file descriptors per process... PASSED
Checking OS max open file descriptors system-wide... PASSED
Checking OS file system and Docker storage driver compatibility... PASSED
-- Completed Verifying Prerequisites --

- Running Bootstrap container
- Monitoring bootstrap process
- [2018-10-01 18:02:29,518][INFO ][no.found.bootstrap.BootstrapInitial] Loaded bootstrap settings {}
- [2018-10-01 18:02:31,847][WARN ][no.found.docker.DockerContainerManager] Default registry [docker.elastic.co] has no auths. Known auths: [List()] {}
- [2018-10-01 18:02:33,169][INFO ][no.found.docker.DockerContainerManager] Starting container [frc-zookeeper-servers-zookeeper] {"ec_container_kind":"docker","ec_container_group":"zookeeper-servers","ec_container_name":"zookeeper"}
- [2018-10-01 18:02:33,179][INFO ][no.found.docker.DockerContainerManager] Starting container [frc-client-forwarders-client-forwarder] {"ec_container_kind":"docker","ec_container_group":"client-forwarders","ec_container_name":"client-forwarder"}
- [2018-10-01 18:02:33,557][INFO ][org.apache.curator.framework.imps.CuratorFrameworkImpl] Starting {}
- [2018-10-01 18:02:33,576][INFO ][no.found.curator.ForwardedEnsembleProvider] Unable to read servers list from [http://172.16.10.130:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1], falling back to [0.0.0.0:2181] {}
- [2018-10-01 18:02:33,577][INFO ][no.found.curator.ForwardedEnsembleProvider] Resolved connection string from [http://172.16.10.130:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1] to [0.0.0.0:2181/v1] with local namespace [/v1] {}
- [2018-10-01 18:02:33,606][INFO ][no.found.curator.ForwardedEnsembleProvider] Unable to read servers list from [http://172.16.10.130:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1], falling back to [0.0.0.0:2181] {}
- [2018-10-01 18:02:33,606][INFO ][no.found.curator.ForwardedEnsembleProvider] Resolved connection string from [http://172.16.10.130:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1] to [0.0.0.0:2181/v1] with local namespace [/v1] {}
- [2018-10-01 18:02:33,608][INFO ][org.apache.curator.framework.imps.CuratorFrameworkImpl] Curator Schemas:
-
-  {}
- [2018-10-01 18:02:34,647][INFO ][no.found.curator.ForwardedEnsembleProvider] Unable to read servers list from [http://172.16.10.130:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1], falling back to [0.0.0.0:2181] {}
- [2018-10-01 18:02:34,648][INFO ][no.found.curator.ForwardedEnsembleProvider] Resolved connection string from [http://172.16.10.130:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1] to [0.0.0.0:2181/v1] with local namespace [/v1] {}
- [2018-10-01 18:02:35,649][INFO ][no.found.curator.ForwardedEnsembleProvider] Unable to read servers list from [http://172.16.10.130:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1], falling back to [0.0.0.0:2181] {}
- [2018-10-01 18:02:35,650][INFO ][no.found.curator.ForwardedEnsembleProvider] Resolved connection string from [http://172.16.10.130:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1] to [0.0.0.0:2181/v1] with local namespace [/v1] {}
- [2018-10-01 18:02:36,651][INFO ][no.found.curator.ForwardedEnsembleProvider] Unable to read servers list from [http://172.16.10.130:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1], falling back to [0.0.0.0:2181] {}
- [2018-10-01 18:02:36,651][INFO ][no.found.curator.ForwardedEnsembleProvider] Resolved connection string from [http://172.16.10.130:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1] to [0.0.0.0:2181/v1] with local namespace [/v1] {}
- [2018-10-01 18:02:37,652][INFO ][no.found.curator.ForwardedEnsembleProvider] Unable to read servers list from [http://172.16.10.130:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1], falling back to [0.0.0.0:2181] {}
- [2018-10-01 18:02:37,652][INFO ][no.found.curator.ForwardedEnsembleProvider] Resolved connection string from [http://172.16.10.130:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1] to [0.0.0.0:2181/v1] with local namespace [/v1] {}
- [2018-10-01 18:02:38,653][INFO ][no.found.curator.ForwardedEnsembleProvider] Unable to read servers list from [http://172.16.10.130:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1], falling back to [0.0.0.0:2181] {}
- [2018-10-01 18:02:38,653][INFO ][no.found.curator.ForwardedEnsembleProvider] Resolved connection string from [http://172.16.10.130:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1] to [0.0.0.0:2181/v1] with local namespace [/v1] {}
- [2018-10-01 18:02:39,654][INFO ][no.found.curator.ForwardedEnsembleProvider] Unable to read servers list from [http://172.16.10.130:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1], falling back to [0.0.0.0:2181] {}
3 
- [2018-10-01 18:02:39,654][INFO ][no.found.curator.ForwardedEnsembleProvider] Resolved connection string from [http://172.16.10.130:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1] to [0.0.0.0:2181/v1] with local namespace [/v1] {}
- [2018-10-01 18:02:40,655][INFO ][no.found.curator.ForwardedEnsembleProvider] Unable to read servers list from [http://172.16.10.130:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1], falling back to [0.0.0.0:2181] {}
- [2018-10-01 18:02:40,655][INFO ][no.found.curator.ForwardedEnsembleProvider] Resolved connection string from [http://172.16.10.130:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1] to [0.0.0.0:2181/v1] with local namespace [/v1] {}
- [2018-10-01 18:02:41,656][INFO ][no.found.curator.ForwardedEnsembleProvider] Unable to read servers list from [http://172.16.10.130:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1], falling back to [0.0.0.0:2181] {}
- [2018-10-01 18:02:41,656][INFO ][no.found.curator.ForwardedEnsembleProvider] Resolved connection string from [http://172.16.10.130:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1] to [0.0.0.0:2181/v1] with local namespace [/v1] {}
- [2018-10-01 18:02:42,657][INFO ][no.found.curator.ForwardedEnsembleProvider] Unable to read servers list from [http://172.16.10.130:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1], falling back to [0.0.0.0:2181] {}
- [2018-10-01 18:02:42,657][INFO ][no.found.curator.ForwardedEnsembleProvider] Resolved connection string from [http://172.16.10.130:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1] to [0.0.0.0:2181/v1] with local namespace [/v1] {}
- [2018-10-01 18:02:43,658][INFO ][no.found.curator.ForwardedEnsembleProvider] Unable to read servers list from [http://172.16.10.130:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1], falling back to [0.0.0.0:2181] {}
- [2018-10-01 18:02:43,658][INFO ][no.found.curator.ForwardedEnsembleProvider] Resolved connection string from [http://172.16.10.130:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1] to [0.0.0.0:2181/v1] with local namespace [/v1] {}
- [2018-10-01 18:02:44,659][INFO ][no.found.curator.ForwardedEnsembleProvider] Unable to read servers list from [http://172.16.10.130:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1], falling back to [0.0.0.0:2181] {}
- [2018-10-01 18:02:44,659][INFO ][no.found.curator.ForwardedEnsembleProvider] Resolved connection string from [http://172.16.10.130:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1] to [0.0.0.0:2181/v1] with local namespace [/v1] {}
- [2018-10-01 18:02:45,660][INFO ][no.found.curator.ForwardedEnsembleProvider] Unable to read servers list from [http://172.16.10.130:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1], falling back to [0.0.0.0:2181] {}
- [2018-10-01 18:02:45,660][INFO ][no.found.curator.ForwardedEnsembleProvider] Resolved connection string from [http://172.16.10.130:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1] to [0.0.0.0:2181/v1] with local namespace [/v1] {}
- [2018-10-01 18:02:46,661][INFO ][no.found.curator.ForwardedEnsembleProvider] Unable to read servers list from [http://172.16.10.130:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1], falling back to [0.0.0.0:2181] {}
- [2018-10-01 18:02:46,661][INFO ][no.found.curator.ForwardedEnsembleProvider] Resolved connection string from [http://172.16.10.130:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1] to [0.0.0.0:2181/v1] with local namespace [/v1] {}
- [2018-10-01 18:02:47,663][INFO ][no.found.curator.ForwardedEnsembleProvider] Unable to read servers list from [http://172.16.10.130:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1], falling back to [0.0.0.0:2181] {}
- [2018-10-01 18:02:47,663][INFO ][no.found.curator.ForwardedEnsembleProvider] Resolved connection string from [http://172.16.10.130:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1] to [0.0.0.0:2181/v1] with local namespace [/v1] {}
- [2018-10-01 18:02:48,914][INFO ][no.found.curator.ForwardedEnsembleProvider] Resolved connection string from [http://172.16.10.130:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1] to [172.16.10.130:22191/v1] with local namespace [/v1] {}
- [2018-10-01 18:02:48,915][INFO ][org.apache.curator.ConnectionState] Connection string changed to: 0.0.0.0:2181/v1 {}
- [2018-10-01 18:02:49,132][INFO ][no.found.curator.ForwardedEnsembleProvider] Resolved connection string from [http://172.16.10.130:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1] to [172.16.10.130:22191/v1] with local namespace [/v1] {}
- [2018-10-01 18:02:49,170][INFO ][no.found.curator.ForwardedEnsembleProvider] Resolved connection string from [http://172.16.10.130:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1] to [172.16.10.130:22191/v1] with local namespace [/v1] {}
- [2018-10-01 18:02:49,171][INFO ][org.apache.curator.framework.state.ConnectionStateManager] State change: CONNECTED {}
- [2018-10-01 18:02:49,193][INFO ][org.apache.curator.framework.imps.EnsembleTracker] New config event received: {version=100000000, server.10=localhost:2898:3898:participant;0.0.0.0:2191} {}
- [2018-10-01 18:02:49,200][INFO ][org.apache.curator.framework.imps.EnsembleTracker] New config event received: {version=100000000, server.10=localhost:2898:3898:participant;0.0.0.0:2191} {}
- [2018-10-01 18:02:49,222][INFO ][org.apache.curator.framework.imps.CuratorFrameworkImpl] Starting {}
- [2018-10-01 18:02:49,225][INFO ][no.found.curator.ForwardedEnsembleProvider] Resolved connection string from [http://172.16.10.130:2180/zookeeper/clients/ensemble/connection-string?namespace=/] to [172.16.10.130:22191/] with local namespace [/] {}
- [2018-10-01 18:02:49,228][INFO ][no.found.curator.ForwardedEnsembleProvider] Resolved connection string from [http://172.16.10.130:2180/zookeeper/clients/ensemble/connection-string?namespace=/] to [172.16.10.130:22191/] with local namespace [/] {}
- [2018-10-01 18:02:49,228][INFO ][org.apache.curator.framework.imps.CuratorFrameworkImpl] Curator Schemas:
-
-  {}
- [2018-10-01 18:02:49,239][INFO ][no.found.curator.ForwardedEnsembleProvider] Resolved connection string from [http://172.16.10.130:2180/zookeeper/clients/ensemble/connection-string?namespace=/] to [172.16.10.130:22191/] with local namespace [/] {}
- [2018-10-01 18:02:49,239][INFO ][org.apache.curator.framework.state.ConnectionStateManager] State change: CONNECTED {}
- [2018-10-01 18:02:49,242][INFO ][org.apache.curator.framework.imps.EnsembleTracker] New config event received: {version=100000000, server.10=localhost:2898:3898:participant;0.0.0.0:2191} {}
4 
- [2018-10-01 18:02:49,242][INFO ][org.apache.curator.framework.imps.EnsembleTracker] New config event received: {version=100000000, server.10=localhost:2898:3898:participant;0.0.0.0:2191} {}
- [2018-10-01 18:02:49,256][INFO ][org.apache.curator.framework.imps.CuratorFrameworkImpl] backgroundOperationsLoop exiting {}
- [2018-10-01 18:02:49,575][INFO ][no.found.bootstrap.BootstrapInitial] Creating container set [runners] with acls: List(31,s{'digest,'runners:LK0Ce6oidMXqEMrqtSjiCvGZWUs=}
- , 31,s{'digest,'runners:rpq8xH41nwCOAeu+jnQXxIqPuTo=}
- ) {}
- [2018-10-01 18:02:49,613][INFO ][no.found.bootstrap.BootstrapInitial] Creating container set [zookeeper-servers] with acls: List(31,s{'digest,'runners:BuzB0CRzI842kPcYO+kWYJeP5bk=}
- , 31,s{'digest,'zookeeper-servers:d0oZqVplOnpVMxWTo+GahnNJIXY=}
- ) {}
- [2018-10-01 18:02:49,623][INFO ][no.found.bootstrap.BootstrapInitial] Creating container set [client-forwarders] with acls: List(31,s{'digest,'runners:BbKP0RmBnjRtWT4JwuAfW8/VPIY=}
- , 31,s{'digest,'client-forwarders:Ptmcud4HUlNwcecpjaGLKHT04Ks=}
- ) {}
- [2018-10-01 18:02:49,635][INFO ][no.found.bootstrap.BootstrapInitial] Creating container set [client-observers] with acls: List(31,s{'digest,'runners:6uLLrHoFnphlB+f/cm00w7WnuQE=}
- , 31,s{'digest,'client-observers:KLrmFuSB7ANmEsNXAB8iMIxTfkU=}
- ) {}
- [2018-10-01 18:02:49,649][INFO ][no.found.bootstrap.BootstrapInitial] Creating container set [directors] with acls: List(31,s{'digest,'runners:aQR+AFcrVbfzSQZ5B+wNZa3WjJA=}
- , 31,s{'digest,'directors:IV18wbrLFnnkoNjWTD67JUPKlBI=}
- ) {}
- [2018-10-01 18:02:49,665][INFO ][no.found.bootstrap.BootstrapInitial] Creating container set [blueprints] with acls: List(31,s{'digest,'runners:F+vpzxG4bJLc1RiwOhNPsJVSHh0=}
- , 31,s{'digest,'blueprints:eHwGbS3NptC6m4n9CFgZiSyb3bc=}
- ) {}
- [2018-10-01 18:02:49,696][INFO ][no.found.bootstrap.BootstrapInitial] Creating container set [services-forwarders] with acls: List(31,s{'digest,'runners:QUncw5u7fIHE+9EwSbdJJznSJxQ=}
- , 31,s{'digest,'services-forwarders:slQuiklAfHscXot3bdlCDVQYDWc=}
- ) {}
- [2018-10-01 18:02:49,722][INFO ][no.found.bootstrap.BootstrapInitial] Creating container set [constructors] with acls: List(31,s{'digest,'runners:qZOjK7gftdP46GqiFYo9uSEvK/s=}
- , 31,s{'digest,'constructors:C2LDNHRKC++oOYSfF3AP8IUOsVM=}
- ) {}
- [2018-10-01 18:02:49,732][INFO ][no.found.bootstrap.BootstrapInitial] Creating container set [allocators] with acls: List(31,s{'digest,'runners:OW9SuENHzOA+REgjcQYOOHHo1Jk=}
- , 31,s{'digest,'allocators:8smJrZPHd7uWoJjaTvG1OaE9joI=}
- ) {}
- [2018-10-01 18:02:49,747][INFO ][no.found.bootstrap.BootstrapInitial] Creating container set [admin-consoles] with acls: List(31,s{'digest,'runners:5eqNXcuCy76bGn98TdZ7yPn2efw=}
- , 31,s{'digest,'admin-consoles:cImilIJ52iwa2Iyrag75M3AYZZc=}
- ) {}
- [2018-10-01 18:02:49,993][INFO ][no.found.bootstrap.BootstrapInitial] Creating container set [cloud-uis] with acls: List(31,s{'digest,'runners:MtVHxs6ZLTC817SYr2Qqmf6YCwA=}
- , 31,s{'digest,'cloud-uis:h305t0PNJPclE++Hhj6e9IzL4vY=}
- ) {}
- [2018-10-01 18:02:50,005][INFO ][no.found.bootstrap.BootstrapInitial] Creating container set [proxies] with acls: List(31,s{'digest,'runners:rStQvR7NWBDuOYMDgh4/V2B52do=}
- , 31,s{'digest,'proxies:RUeEyJD2493MFhhHLlpNHraZH1s=}
- ) {}
- [2018-10-01 18:02:50,022][INFO ][no.found.bootstrap.BootstrapInitial] Creating container set [beats-runners] with acls: List(31,s{'digest,'runners:TstMojHYjMq6crjMI+ENxKThJNQ=}
- , 31,s{'digest,'beats-runners:x8BA9fdXS/s7WItkTy0UqJHryjk=}
- ) {}
- [2018-10-01 18:02:50,037][INFO ][no.found.bootstrap.BootstrapInitial] Creating container set [curators] with acls: List(31,s{'digest,'runners:xmUcjb92MCig9yOwZB1WeObdHss=}
- , 31,s{'digest,'curators:mQ2l1xKL50hGc+z34mPw7ttf2OQ=}
- ) {}
- [2018-10-01 18:02:50,076][INFO ][no.found.featureflags.staterepository.ZookeeperStateRepository] Initialized cache for feature flag listener {}
- [2018-10-01 18:02:50,098][INFO ][no.found.bootstrap.BootstrapInitial] Initialising feature flag [DedicatedNodeTypes] to [true] {}
- [2018-10-01 18:02:50,102][INFO ][no.found.bootstrap.BootstrapInitial] Starting local runner {}
- [2018-10-01 18:02:50,127][WARN ][no.found.featureflags.staterepository.ZookeeperStateRepository] Feature flag [DedicatedNodeTypes] state does not parse: [], resetting feature to default state {}
- [2018-10-01 18:02:50,135][INFO ][no.found.docker.DockerContainerManager] Starting container [frc-runners-runner] {"ec_container_kind":"docker","ec_container_group":"runners","ec_container_name":"runner"}
- [2018-10-01 18:02:50,223][INFO ][no.found.featureflags.staterepository.ZookeeperStateRepository] Feature flag updated: [DedicatedNodeTypes] with [FeatureState(true,None,None)] {}
- [2018-10-01 18:02:50,243][INFO ][no.found.bootstrap.BootstrapInitial] Started local runner {}
- [2018-10-01 18:02:50,246][INFO ][no.found.bootstrap.BootstrapInitial] Waiting for runner container node {}
- [2018-10-01 18:02:56,466][INFO ][no.found.bootstrap.BootstrapInitial] Runner container node detected {}
- [2018-10-01 18:02:56,637][INFO ][no.found.bootstrap.BootstrapInitial] Waiting for coordinator candidate {}
- [2018-10-01 18:03:11,470][INFO ][no.found.bootstrap.BootstrapInitial] Detected coordinator candidate {}
- [2018-10-01 18:03:11,480][INFO ][no.found.bootstrap.BootstrapInitial] Detected pending coordinator, promoting coordinator {}
- [2018-10-01 18:03:11,582][INFO ][no.found.zookeeper.models.coordinators.CoordinatorCandidatePendingInfo] Current instances: [4294967329,4294967329,1538416969378,1538416969378,0,0,0,0,2,0,4294967329], secrets: [4294967331,4294967331,1538416969387,1538416969387,0,0,0,0,20062,0,4294967331] {}
- [2018-10-01 18:03:11,699][INFO ][no.found.bootstrap.BootstrapInitial] Coordinator accepted {}
- [2018-10-01 18:03:11,701][INFO ][no.found.bootstrap.BootstrapInitial] Storing current platform version: 2.0.0 {}
- [2018-10-01 18:03:11,770][INFO ][no.found.bootstrap.BootstrapInitial] Storing Instance Types: [elasticsearch,kibana] {}
- [2018-10-01 18:03:11,936][INFO ][no.found.bootstrap.BootstrapInitial] Storing Elastic Stack versions: [5.6.12,6.4.1] {}
- [2018-10-01 18:03:12,893][INFO ][no.found.bootstrap.BootstrapInitial] Creating Admin Console Elasticsearch backend {}
- [2018-10-01 18:03:13,636][INFO ][no.found.bootstrap.ServiceLayerBootstrap] Waiting for [ensuring-plan] to complete. Retrying every [1 second] (cause: [org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /clusters/6822135e015d46489eb100ba46663873/plans/status]) {}
5 
- [2018-10-01 18:03:29,515][INFO ][no.found.bootstrap.ServiceLayerBootstrap] Waiting for [ensuring-plan] to complete. Retrying every [1 second] (cause: [java.lang.Exception: not yet started]) {}
- [2018-10-01 18:03:40,833][INFO ][no.found.bootstrap.ServiceLayerBootstrap] Waiting for [ensuring-plan] to complete. Retrying every [1 second] (cause: [org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /clusters/6822135e015d46489eb100ba46663873/plans/pending]) {}
- [2018-10-01 18:03:46,958][INFO ][no.found.bootstrap.BootstrapInitial] Applying Admin Console Elasticsearch index templates {}
- [2018-10-01 18:03:47,010][INFO ][no.found.bootstrap.ServiceLayerBootstrap] Waiting for [apply-elasticsearch-template] to complete. Retrying every [1 second] (cause: [java.lang.Exception: 410 Gone: HttpEntity(application/json; charset=UTF-8,{"ok":false,"message":"Deleted cluster."})]) {}
- [2018-10-01 18:03:49,048][INFO ][no.found.bootstrap.ServiceLayerBootstrap] Waiting for [apply-elasticsearch-template] to complete. Retrying every [1 second] (cause: [java.lang.Exception: 502 Bad Gateway: HttpEntity(application/json; charset=UTF-8,{"ok":false,"message":"The requested cluster is currently unavailable"})]) {}
- [2018-10-01 18:04:04,345][INFO ][no.found.bootstrap.ServiceLayerBootstrap] Waiting for [apply-elasticsearch-template] to complete. Retrying every [1 second] (cause: [java.lang.Exception: 410 Gone: HttpEntity(application/json; charset=UTF-8,{"ok":false,"message":"Deleted cluster."})]) {}
- [2018-10-01 18:13:46,963][ERROR][no.found.bootstrap.BootstrapInitial$] Unhandled error. {}
-- An error has occurred in bootstrap process. Please examine logs --
- java.util.concurrent.TimeoutException: Futures timed out after [600000 milliseconds]
-       at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:223)
-       at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:227)
-       at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:190)
-       at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
-       at scala.concurrent.Await$.result(package.scala:190)
-       at no.found.bootstrap.BootstrapInitial.bootstrapServiceLayer(BootstrapInitial.scala:963)
-       at no.found.bootstrap.BootstrapInitial.bootstrap(BootstrapInitial.scala:678)
-       at no.found.bootstrap.BootstrapInitial$.delayedEndpoint$no$found$bootstrap$BootstrapInitial$1(BootstrapInitial.scala:1311)
-       at no.found.bootstrap.BootstrapInitial$delayedInit$body.apply(BootstrapInitial.scala:1305)
-       at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
-       at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
-       at scala.App$$anonfun$main$1.apply(App.scala:76)
-       at scala.App$$anonfun$main$1.apply(App.scala:76)
-       at scala.collection.immutable.List.foreach(List.scala:392)
-       at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
-       at scala.App$class.main(App.scala:76)
-       at no.found.bootstrap.BootstrapInitial$.main(BootstrapInitial.scala:1305)
-       at no.found.bootstrap.BootstrapInitial.main(BootstrapInitial.scala)
- [2018-10-01 18:13:47,003][INFO ][no.found.bootstrap.ServiceLayerBootstrap] Waiting for [apply-elasticsearch-template] to complete. Retrying every [1 second] (cause: [akka.pattern.AskTimeoutException: Recipient[Actor[akka://default/user/IO-HTTP#-416208076]] had already been terminated. Sender[null] sent the message of type "spray.http.HttpRequest".]) {}
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  Errors have caused Elastic Cloud Enterprise installation to fail - Please check logs
  Node type - initial
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Sorry for all the replies. I wasn't sure how else to get the debug messages in here.

6

Hi @naknak987,
According to what I see, the bootstrap failed while setting up one of system-owned Elasticsearch clusters. This can happen due to different reasons. To help you, I will need to check logs.
Can you please send me logs through private message? You can use a diagnostic script or just archive entire storage directory (the default is /mnt/data/elastic)

7

@naknak987, thanks for sharing logs. Based on what I see in the logs, I think you did not set up permissions correctly. Please make sure your host meets the prerequisites https://www.elastic.co/guide/en/cloud-enterprise/current/ece-prereqs.html. User elastic must belong to sudo and docker groups.

8

The group sudo does not exist on the system. The user elastic can however sudo.

[root@elsson01 elastic]# cat /etc/sudoers.d/sudoers
%domain\ admins ALL=(ALL)       ALL
%elastic        ALL=(ALL)       ALL

The user elastic does already belong to the docker group.

[root@elsson01 elastic]# id elastic
uid=1000(elastic) gid=1001(elastic) groups=1001(elastic),0(root),991(dockerroot),1000(docker)
9

Hi @naknak987, while checking at the diagnostics I noticed that you are using the latest Kernel 4.4.158 which has a regression on the OverlayFs. This regression was added on 4.4.156 so I would like to ask to use 4.4.155 version instead.

10

@Alexandros_Sapranidi
I'm having trouble finding information on switching to kernel 4.4.155. Would you be able to provide some instructions.

11

You can pick a mirror from http://elrepo.org/tiki/Download, note use the mirrors with the archive text next to them, for example, ftp://ftp.nluug.nl/pub/os/Linux/distr/elrepo/archive/kernel/el7/x86_64/RPMS/
Then you can download the RPM of the kernel and install it, please note that prior to using the kernel you should reboot your system.

1 Like
12

@Alexandros_Sapranidi

Thank you so much! The kernel was the issue. Should the following documentation be updated to use the correct kernel so that future users don't have this issue?

https://www.elastic.co/guide/en/cloud-enterprise/current/ece-configure-hosts.html#ece-configure-hosts-rhel-centos

13

Yes we are working on updating the documentation.
I am glad you managed to install ECE.

1 Like
14

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.