ECE installation hanging after Loaded bootstrap settings

Hi,

I'm installing ECE on RHEL 7.1 in AWS.
I've followed the instructions from here (using the ec2-user) - https://www.elastic.co/guide/en/cloud-enterprise/current/ece-configure-hosts.html#ece-configure-hosts-rhel-centos
(except the Kernel part - it's already 3.10)
When I get to installing ECE it hangs after Loaded bootstrap settings

The issue seems to be the same as the first part of this thread - Issues with installing elastic-cloud-enterprise-installer but it doesn't say what was done to resolve the issue.

Here's the debug output from the install

-- Debug Info --
NODE_TYPE - initial
REPOSITORY - docker.elastic.co/cloud-enterprise
HOST_STORAGE_PATH - /mnt/data/elastic
RUNNER_ROLES -
RUNNER_REGION - ece-region
HOST_STORAGE_DEVICE_PATH - /dev/xvda2
RUNNER_HOST_IP - 172.31.xxx.xxx
HOST_STORAGE_ROOT_VOLUME_PATH - /mnt/data
CLOUD_ENTERPRISE_VERSION - 1.0.2
HOST_PREREQ_FAILED - False
RUNNER_ROLES_TOKEN -
RUNNER_ID - 172.31.xxx.xxx
ELASTIC_GID - 1000
FORCE_INSTALL - False
RUNNER_AVAILABILITY_ZONE - ece-region-1a
RUNNER_CAPACITY - 12843
NODE_TYPE - initial
MEMORY_SETTINGS - {}
ELASTIC_UID - 1000

-- Verifying Prerequisites --
Checking host storage path... PASSED
Checking docker version... PASSED
Checking internal ip connectivity... PASSED
Checking OS settings... PASSED
Checking OS kernel version... PASSED
 - OS kernel version is 3.10.0-229.el7.x86_64 but we recommend 4.4.
Checking Minimum required memory... PASSED
Checking Kernel cgroup.memory setting... PASSED
 - OS setting 'cgroup.memory' should be set to cgroup.memory=nokmem
-- Completed Verifying Prerequisites --

- Running Bootstrap container
- Monitoring bootstrap process
- [2017-10-19 11:31:14,330][INFO ][no.found.bootstrap.BootstrapInitial] Loaded bootstrap settings {}
- [2017-10-19 11:31:18,440][WARN ][no.found.docker.DockerContainerManager] Default registry [https://index.docker.io/v1/] has no auths. Known auths: [List()] {}
- [2017-10-19 11:31:20,251][INFO ][no.found.docker.DockerContainerManager] Starting container [frc-client-forwarders-client-forwarder] {"ec_container_kind":"docker","ec_container_group":"client-forwarders","ec_container_name":"client-forwarder"}
- [2017-10-19 11:31:20,272][INFO ][no.found.docker.DockerContainerManager] Starting container [frc-zookeeper-servers-zookeeper] {"ec_container_kind":"docker","ec_container_group":"zookeeper-servers","ec_container_name":"zookeeper"}
- [2017-10-19 11:31:20,889][INFO ][org.apache.curator.framework.imps.CuratorFrameworkImpl] Starting {}
- [2017-10-19 11:31:20,904][INFO ][no.found.curator.ForwardedEnsembleProvider] Unable to read servers list from [http://172.31.xxx.xxx:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1], falling back to [0.0.0.0:2181] {}
- [2017-10-19 11:31:20,905][INFO ][no.found.curator.ForwardedEnsembleProvider] Resolved connection string from [http://172.31.xxx.xxx:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1] to [0.0.0.0:2181/v1] with local namespace [/v1] {}
- [2017-10-19 11:31:20,928][INFO ][no.found.curator.ForwardedEnsembleProvider] Unable to read servers list from [http://172.31.xxx.xxx:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1], falling back to [0.0.0.0:2181] {}
- [2017-10-19 11:31:20,928][INFO ][no.found.curator.ForwardedEnsembleProvider] Resolved connection string from [http://172.31.xxx.xxx:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1] to [0.0.0.0:2181/v1] with local namespace [/v1] {}
 - [2017-10-19 11:31:20,929][INFO ][org.apache.curator.framework.imps.CuratorFrameworkImpl] Default schema {}
- [2017-10-19 11:31:21,930][INFO ][no.found.curator.ForwardedEnsembleProvider] Unable to read servers list from [http://172.31.xxx.xxx:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1], falling back to [0.0.0.0:2181] {}
- [2017-10-19 11:31:21,931][INFO ][no.found.curator.ForwardedEnsembleProvider] Resolved connection string from [http://172.31.xxx.xxx:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1] to [0.0.0.0:2181/v1] with local namespace [/v1] {}
- [2017-10-19 11:31:22,932][INFO ][no.found.curator.ForwardedEnsembleProvider] Unable to read servers list from [http://172.31.xxx.xxx:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1], falling back to [0.0.0.0:2181] {}
- [2017-10-19 11:31:22,932][INFO ][no.found.curator.ForwardedEnsembleProvider] Resolved connection string from [http://172.31.xxx.xxx:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1] to [0.0.0.0:2181/v1] with local namespace [/v1] {}
- [2017-10-19 11:31:23,933][INFO ][no.found.curator.ForwardedEnsembleProvider] Unable to read servers list from [http://172.31.xxx.xxx:2180/zookeeper/clients/ensemble/connection-string?namespace=/v1], falling back to [0.0.0.0:2181] {}

if it helps, here's the output of docker ps at this point

CONTAINER ID        IMAGE                                                               COMMAND                  CREATED              STATUS              PORTS                                                                                                  NAMES
9752d15082d4        docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.0.2   "/sbin/my_init"          About a minute ago   Up About a minute   0.0.0.0:2191->2191/tcp, 0.0.0.0:12191->12191/tcp, 0.0.0.0:12898->12898/tcp, 0.0.0.0:13898->13898/tcp   frc-zookeeper-servers-zookeeper
960c5798418c        docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.0.2   "/sbin/my_init"          About a minute ago   Up About a minute                                                                                                          frc-client-forwarders-client-forwarder
d59ba3e6dbf9        docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.0.2   "/sbin/my_init"          About a minute ago   Up About a minute                                                                                                          elastic-cloud-enterprise-bootstrap-1.0.2
d217d9047e9f        docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.0.2   "elastic-cloud-enterp"   About a minute ago   Up About a minute   0.0.0.0:20000->20000/tcp                                                                               elastic-cloud-enterprise-installer-1.0.2

Hi @adesouza

Sorry you're having issues. It looks like the installer is unable to connect to the frc-zookeeper-servers-zookeeper, which occurs via the frc-client-forwarders-client-forwarder (on port 2180) (which encrypts everything)

The fix for the similar issue you linked was in this comment:

This looks like a connectivity issue. Is the public hostname you specified at the beginning of the installation routable from within the host? If you're on AWS for example, you should use the AWS public hostname and not public ip.

Your case seems different, am I right that you're in a VPC and specified the internal VPC address as the host?

A couple of possibilities:

  • Is the RUNNER_IP (172.32.*) definitely right? Pretty sure the installer would fail our earlier if not, but worth double checking
  • You should have a port open on 2180 - do you have some strange iptables configuration that is blocking it for the local IP?
    • (If he 2180 port is open, can you connect to it from the host CLI?)
  • Can you share the logs from frc-client-forwarders-client-forwarder (you should be able to find them in /mnt/data/elastic/RUNNERIP/services/client-forwarder/logs or somewhere like that)

It's definitely a slightly strange networking problem, anyway

Alex

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.