Installation if ECE beta1 fails on RHEL

Elastic Orchestration Elastic Cloud Enterprise (ECE)
1

Hello. I am trying to install elastic cloud beta1 using offline installation method, but it always fails with the following:

- [2017-03-14 07:36:56,032][INFO ][no.found.bootstrap.BootstrapInitial] Applying Elasticsearch index template {}
- [2017-03-14 07:36:56,269][INFO ][no.found.bootstrap.BootstrapInitial] Updating dynamic Admin Console settings with Elasticsearch cluster information {}
- [2017-03-14 07:36:56,281][INFO ][no.found.bootstrap.BootstrapInitial] Starting reindexing Admin Console data {}
- [2017-03-14 07:36:56,714][INFO ][no.found.bootstrap.BootstrapInitial] Creating Logging and Metrics Cluster {}
- [2017-03-14 07:36:56,828][INFO ][no.found.bootstrap.ServiceLayerBootstrap] Waiting for [ensuring-plan] to complete. Retrying every [1 second] (cause: [org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /clusters/97e9d02566fa4cb1b1fb26bba49f168c/plans/status]) {}
- [2017-03-14 07:36:57,850][INFO ][no.found.bootstrap.ServiceLayerBootstrap] Waiting for [ensuring-plan] to complete. Retrying every [1 second] (cause: [java.lang.Exception: not yet started]) {}

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Errors have caused Elastic Cloud Enterprise installation to fail

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Error response from daemon: Unable to remove filesystem for 4a05e35c00f6db3ab39ce14e8fd72b2d8fc4c05ce042d40508dabc1714d94437: remove /var/lib/docker/containers/4a05e35c00f6db3ab39ce14e8fd72b2d8fc4c05ce042d40508dabc1714d94437/shm: device or resource busy

I am installing as elastic user. Elastic user is member of docker group, ... the environment is prepared as recommended in https://www.elastic.co/guide/en/cloud-enterprise/current/ece-configure-hosts.html

RHEL release: 7.3 (Maipo), selinux disabled

docker images

REPOSITORY                                                    TAG                 IMAGE ID            CREATED             SIZE
docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise   1.0.0-beta1         9fdf5181c9ce        12 days ago         549.8 MB
docker.elastic.co/cloud-enterprise/elasticsearch              5.2.0-1             14d18b0676ff        4 weeks ago         1.169 GB
docker.elastic.co/cloud-enterprise/elasticsearch              2.4.4-1             f19d395f6016        5 weeks ago         914.4 MB
docker.elastic.co/cloud-enterprise/kibana                     4.6.4               72254a092199        5 weeks ago         1.063 GB
docker.elastic.co/cloud-enterprise/kibana                     5.2.0               a6b1e3ece095        5 weeks ago         830.8 MB

docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d8e5909f9e5f docker.elastic.co/cloud-enterprise/elasticsearch:2.4.4-1 "/sbin/entry-point" 26 minutes ago Up 26 minutes 0.0.0.0:18356->18356/tcp, 0.0.0.0:19020->19020/tcp fac-83522813bacd488f86708fe9ed381e5f-instance-0000000000
fa15d68f1687 docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.0.0-beta1 "/sbin/my_init" 27 minutes ago Up 26 minutes 0.0.0.0:8764->8764/tcp, 0.0.0.0:12300->12300/tcp, 0.0.0.0:12343->12343/tcp frc-admin-consoles-admin-console
469888d53edb docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.0.0-beta1 "/sbin/my_init" 27 minutes ago Up 26 minutes 0.0.0.0:12400->5601/tcp, 0.0.0.0:12443->5643/tcp frc-cloud-uis-cloud-ui
fae4022c80d6 docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.0.0-beta1 "/sbin/my_init" 27 minutes ago Up 27 minutes frc-constructors-constructor
ee4aab161e60 docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.0.0-beta1 "/sbin/my_init" 27 minutes ago Up 27 minutes frc-services-forwarders-services-forwarder
defbe31703d5 docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.0.0-beta1 "/sbin/my_init" 27 minutes ago Up 27 minutes 0.0.0.0:2112->2112/tcp frc-directors-director
d30cbdc0f22c docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.0.0-beta1 "/sbin/my_init" 27 minutes ago Up 27 minutes frc-beats-runners-beats-runner
4705844b9660 docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.0.0-beta1 "/sbin/my_init" 27 minutes ago Up 27 minutes frc-allocators-allocator
708eba63c9a3 docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.0.0-beta1 "/sbin/my_init" 27 minutes ago Up 27 minutes 0.0.0.0:9200->9200/tcp, 0.0.0.0:9243->9243/tcp, 0.0.0.0:9300->9300/tcp, 0.0.0.0:9343->9343/tcp frc-proxies-proxy
bbb26c2ac883 docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.0.0-beta1 "/sbin/my_init" 27 minutes ago Up 27 minutes frc-blueprints-blueprint
5dc334f20383 docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.0.0-beta1 "/sbin/my_init" 27 minutes ago Up 27 minutes frc-runners-runner
d51fb0335abb docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.0.0-beta1 "/sbin/my_init" 27 minutes ago Up 27 minutes frc-client-forwarders-client-forwarder
6096cc9379f5 docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.0.0-beta1 "/sbin/my_init" 27 minutes ago Up 27 minutes 0.0.0.0:2191->2191/tcp, 0.0.0.0:12191->12191/tcp, 0.0.0.0:12898->12898/tcp, 0.0.0.0:13898->13898/tcp frc-zookeeper-servers-zookeeper

curl -H 'x-found-cluster: 97e9d02566fa4cb1b1fb26bba49f168c' 'localhost:9200'

{"ok":false,"message":"The requested cluster is currently unavailable"}

2

BTW. I tried this during the installation process before the "fails message"

[root@saaa ~]# curl -H 'x-found-cluster: aa411c404c014461bc2c6d3b848bc39a' 'localhost:9200'
{
  "name" : "instance-0000000000",
  "cluster_name" : "aa411c404c014461bc2c6d3b848bc39a",
  "cluster_uuid" : "9jTu-9xvRcKz88JsMuWhmw",
  "version" : {
    "number" : "2.4.4",
    "build_hash" : "fcbb46dfd45562a9cf00c604b30849a6dec6b017",
    "build_timestamp" : "2017-01-03T11:33:16Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.2"
  },
  "tagline" : "You Know, for Search"
}
[root@saaa ~]# curl -H 'x-found-cluster: aa411c404c014461bc2c6d3b848bc39a' 'localhost:9244'
{
  "name" : "instance-0000000000",
  "cluster_name" : "aa411c404c014461bc2c6d3b848bc39a",
  "cluster_uuid" : "9jTu-9xvRcKz88JsMuWhmw",
  "version" : {
    "number" : "2.4.4",
    "build_hash" : "fcbb46dfd45562a9cf00c604b30849a6dec6b017",
    "build_timestamp" : "2017-01-03T11:33:16Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.2"
  },
  "tagline" : "You Know, for Search"
}
[root@saaa ~]# curl -H 'x-found-cluster: aa411c404c014461bc2c6d3b848bc39a' 'localhost:18025'
curl: (7) Failed connect to localhost:18025; Connection refused
3

OK, I found the issue in allocator log. It tries to pull elasticsearch 5.2.0-2, I downloaded 5.2.0-1.

4

@vb4t,

I'm glad you were able to figure out what the issue is. I think the current offline instructions we put together list all the available images, but I think there might be a usability issue around the different build versions indicated by a dash. I will have a chat with our developers working on the installation script to see if this is working as designed or if there is an easier way to indicate what build versions you need without having to hunt through the logs.

Nik

5

Hi @vb4t,

The offline installation is being improved for 1.0 GA. This should see more of the manual operations currently needed in beta 1 to be part of the installation script. For now, glad you were able to find the issue and we will look at the documentation to see if we can improve it for beta 1 users.

Christian

6

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.