ECE installation hanging on bootstrap issue

Elastic Orchestration Elastic Cloud Enterprise (ECE)
1

Hi,

I m' very interested in ECE capbilities. Currently trying to install it to figure out exactly what it actually does, and does not.
For now, I'm stuck in the installation process, hanging on the "Monitoring bootstrap process":

sh elastic-cloud-enterprise.sh install --debug

-- Debug Info --
NODE_TYPE - initial
REPOSITORY - docker.elastic.co/cloud-enterprise
HOST_STORAGE_DEVICE_PATH - /dev/mapper/vg0-datalv
ELASTIC_UID - 0
RUNNER_ROLES_TOKEN -
MEMORY_SETTINGS - {}
CLOUD_ENTERPRISE_VERSION - 1.0.1
ELASTIC_GID - 0
RUNNER_HOST_IP -
NODE_TYPE - initial
RUNNER_AVAILABILITY_ZONE - ece-region-1a
RUNNER_REGION - ece-region
RUNNER_ROLES -
HOST_STORAGE_PATH - /data/elastic
RUNNER_CAPACITY - 13204
HOST_STORAGE_ROOT_VOLUME_PATH - /data
RUNNER_ID -

-- Verifying Prerequisites --
Checking host storage path... PASSED
Checking docker version... PASSED

  • Elastic Cloud Enterprise is certified for 1.11. but 1.12.6 is currently installed
    Checking internal ip connectivity... PASSED
    Checking OS settings... PASSED
    Checking OS kernel version... PASSED
    Checking Minimum required memory... PASSED
    Checking Kernel cgroup.memory setting... PASSED

  • OS setting 'cgroup.memory' should be set to cgroup.memory=nokmem
    -- Completed Verifying Prerequisites --

  • Running Bootstrap container

  • Monitoring bootstrap process
    Process bootstrap logging processor:
    Traceback (most recent call last):
    File "/usr/lib/python3.5/multiprocessing/process.py", line 249, in _bootstrap
    self.run()
    File "/usr/lib/python3.5/multiprocessing/process.py", line 93, in run
    self._target(*self._args, **self._kwargs)
    File "/elastic_cloud_apps/bootstrap-initiator/bootstrap_initiator/logging.py", line 73, in monitor_bootstrap_log
    raise IOError("{0}/logs/bootstrap-logs/bootstrap.log doesn't exist after 1 min of retrying".format(bootstrap_properties['HOST_STORAGE_PATH']))
    OSError: /data/elastic/logs/bootstrap-logs/bootstrap.log doesn't exist after 1 min of retrying
    ^CTraceback (most recent call last):
    File "/elastic_cloud_apps/bootstrap-initiator/initiator.py", line 82, in
    monitor.logging_and_bootstrap_monitor(bootstrap_properties, enable_debug)
    File "/elastic_cloud_apps/bootstrap-initiator/bootstrap_initiator/monitor.py", line 18, in logging_and_bootstrap_monitor
    sleep(5)
    KeyboardInterrupt

Installation is done on centOS7 as root user, as Docker is running as root user: implemented that way in the company I work for (no choice on that...).
I've seen some other topics with the same issue, solved by changing permissions on data repositories and subfolders, but unfortunately it did not help in my case: install still freezing on bootstrap monitoring step...

Any help would be appreciated...

Many thanks in advance!

Aymeric

2

Hi @Aymeric,

Installation as root will fail and we don't support it, as per our official documentation. You will need to install ECE with a non-root user who is part of the docker group.

A future version of ECE will not permit the install to go ahead if the UID and GID are outside of a certain range. That is, the installer will still fail, but it will fail at the stage where we check the prerequisites and well before you even hit the error shown in your post. (I know that doesn't immediately help you, but that's the best I can offer, even after talking to a couple of developers about your question. :slight_smile:)

Nik

3

Hi Nik,

Thanks for your reply. I've so created an Elastic user and run the install again.
It went further but failed again unfortunately:

it failed at the following step:

  • Running Bootstrap container
  • Monitoring bootstrap process
  • Loaded bootstrap settings {}
  • Starting local runner {}
  • Started local runner {}
  • Waiting for runner container node {}
  • Runner container node detected {}
  • Waiting for coordinator candidate {}
  • Detected coordinator candidate {}
  • Detected pending coordinator, promoting coordinator {}
  • Coordinator accepted {}
  • Storing current platform version: 1.0.1 {}
  • Storing Elastic Stack versions: [2.4.5,5.4.1] {}
  • Creating Admin Console Elasticsearch backend {}
  • Unhandled error. {}
    -- An error has occurred in bootstrap process. Please examine logs --
    Exception in thread Thread-4:
    Traceback (most recent call last):
    File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
    File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
    File "/usr/lib/python3.5/site-packages/sh.py", line 1484, in output_thread
    done = stream.read()
    File "/usr/lib/python3.5/site-packages/sh.py", line 1974, in read
    self.write_chunk(chunk)
    File "/usr/lib/python3.5/site-packages/sh.py", line 1949, in write_chunk
    self.should_quit = self.process_chunk(chunk)
    File "/usr/lib/python3.5/site-packages/sh.py", line 1867, in process
    return handler(chunk)
    File "/usr/lib/python3.5/site-packages/sh.py", line 1106, in fn
    return handler(chunk, *args)
    File "/elastic_cloud_apps/bootstrap-initiator/bootstrap_initiator/logging.py", line 34, in process_info_log_output
    puts("- {0}".format(line.split("]",3)[3].strip()))
    IndexError: list index out of range

bootsrap.log was mentionning a 600s timeout. I saw in another post to restrat the install tool, what I did, and now it fails with the following error:

  • Running Bootstrap container
  • Monitoring bootstrap process
  • Loaded bootstrap settings {}
  • Unhandled error. {}
    -- An error has occurred in bootstrap process. Please examine logs --
  Errors have caused Elastic Cloud Enterprise installation to fail - Please check logs
  Node type - initial

In bootstrap.log file:
[2017-06-15 15:23:58,786][INFO ][org.apache.curator.framework.state.ConnectionStateManager] State change: CONNECTED {}
[2017-06-15 15:23:58,801][INFO ][org.apache.curator.framework.imps.EnsembleTracker] New config event received: {version=100000000, server.10=localhost:2898:3898:participant;0.0.0.0:2191} {}
[2017-06-15 15:23:58,802][INFO ][no.found.curator.ForwardedEnsembleProvider] ZooKeeper connection string set to [0.0.0.0:2191] {}
[2017-06-15 15:23:58,806][INFO ][org.apache.curator.framework.imps.CuratorFrameworkImpl] backgroundOperationsLoop exiting {}
[2017-06-15 15:23:58,935][ERROR][no.found.bootstrap.BootstrapInitial$] Unhandled error. {}
org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = NodeExists for /bootstrap/install
at org.apache.zookeeper.KeeperException.create(KeeperException.java:123)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
at no.found.curator.FutureBackgroundCallback.processResult(FutureBackgroundCallback.scala:22)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.sendToBackgroundCallback(CuratorFrameworkImpl.java:835)
at org.apache.curator.framework.imps.CuratorFrameworkImpl.processBackgroundOperation(CuratorFrameworkImpl.java:618)
at org.apache.curator.framework.imps.CreateBuilderImpl.sendBackgroundResponse(CreateBuilderImpl.java:783)
at org.apache.curator.framework.imps.CreateBuilderImpl.access$1300(CreateBuilderImpl.java:44)
at org.apache.curator.framework.imps.CreateBuilderImpl$7.processResult(CreateBuilderImpl.java:623)
at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:672)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:528)

Even after removing all containers (docker rm -f of each containers), it still fails with the same issue.

What is the process for a clean desinstall before reinstall? How to avoid the timeout during install?

Thanks for your help.

Aymeric

4

@Aymeric,

The steps for removing ECE are documented, see https://www.elastic.co/guide/en/cloud-enterprise/current/ece-uninstall.html.

The TL;DR: is that you need to run this command: docker rm -f frc-runners-runner frc-allocators-allocator $(docker ps -a -q); sudo rm -rf /mnt/data/elastic/* && docker ps -a.

I hope this helps,

Nik

5

Hi @Aymeric

The issue that you're seeing is likely that the timeout we specified for various Elasticsearch and Kibana provisioning tasks is too low. You can confirm by looking at /mnt/data/elastic/logs/bootstrap-logs/bootstrap.log, there should be a timeout error. We've seen this a few times now.

We're addressing this for 1.0.2, which will be released in a couple of weeks.

In the meantime, the quickest fix is to remove ECE and run again identically - the docker images stay pulled, so the provisioning is faster next time. (you can also pre-pull the images - this way is probably easier though)

Apologies for the issues!

Alex

6

Hi Nik and Alex,

Thanks for your reply and many thanks for your help.

I'm working with the offline install procedure as servers here do not have Internet access, so the images are already loaded (can see them with "docker images").
I ran the uninstall procedure and I've removed from docker all the containers and images I was having and then I've reloaded the images ("docker load").

Same issue during install at "Creating Admin Console ES backend" step:

  • Unhandled error. {}
    -- An error has occurred in bootstrap process. Please examine logs --
    Exception in thread Thread-4:
    Traceback (most recent call last):
    File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
    self.run()
    File "/usr/lib/python3.5/threading.py", line 862, in run
    self._target(*self._args, **self._kwargs)
    File "/usr/lib/python3.5/site-packages/sh.py", line 1484, in output_thread
    done = stream.read()
    File "/usr/lib/python3.5/site-packages/sh.py", line 1974, in read
    self.write_chunk(chunk)
    File "/usr/lib/python3.5/site-packages/sh.py", line 1949, in write_chunk
    self.should_quit = self.process_chunk(chunk)
    File "/usr/lib/python3.5/site-packages/sh.py", line 1867, in process
    return handler(chunk)
    File "/usr/lib/python3.5/site-packages/sh.py", line 1106, in fn
    return handler(chunk, *args)
    File "/elastic_cloud_apps/bootstrap-initiator/bootstrap_initiator/logging.py", line 34, in process_info_log_output
    puts("- {0}".format(line.split("]",3)[3].strip()))
    IndexError: list index out of range

In bootstrap log file:
[2017-06-16 07:21:09,569][INFO ][no.found.bootstrap.ServiceLayerBootstrap] Waiting for [ensuring-plan] to complete. Retrying every [1 second] (cause: [org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /clusters/db8e9fe095ce4dd3900c936138350084/plans/status]) {}
[2017-06-16 07:21:42,346][INFO ][no.found.bootstrap.ServiceLayerBootstrap] Waiting for [ensuring-plan] to complete. Retrying every [1 second] (cause: [java.lang.Exception: not yet started]) {}
[2017-06-16 07:31:09,416][ERROR][no.found.bootstrap.BootstrapInitial$] Unhandled error. {}
java.util.concurrent.TimeoutException: Futures timed out after [600 seconds]
at scala.concurrent.impl.Promise$DefaultPromise.ready(Promise.scala:219)
at scala.concurrent.impl.Promise$DefaultPromise.result(Promise.scala:223)
at scala.concurrent.Await$$anonfun$result$1.apply(package.scala:190)
at scala.concurrent.BlockContext$DefaultBlockContext$.blockOn(BlockContext.scala:53)
at scala.concurrent.Await$.result(package.scala:190)
at no.found.bootstrap.BootstrapInitial.bootstrapServiceLayer(BootstrapInitial.scala:828)
at no.found.bootstrap.BootstrapInitial.bootstrap(BootstrapInitial.scala:609)
at no.found.bootstrap.BootstrapInitial$.delayedEndpoint$no$found$bootstrap$BootstrapInitial$1(BootstrapInitial.scala:1153)
at no.found.bootstrap.BootstrapInitial$delayedInit$body.apply(BootstrapInitial.scala:1147)
at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
at scala.App$$anonfun$main$1.apply(App.scala:76)
at scala.App$$anonfun$main$1.apply(App.scala:76)
at scala.collection.immutable.List.foreach(List.scala:381)
at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
at scala.App$class.main(App.scala:76)
at no.found.bootstrap.BootstrapInitial$.main(BootstrapInitial.scala:1147)
at no.found.bootstrap.BootstrapInitial.main(BootstrapInitial.scala)

Don't know what to do to avoid this timeout issue...

Aymeric
Many thanks again for your help!

7

However, sveral containers were successfully created:
$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
8fd2fddecb70 docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.0.1 "/sbin/my_init" 12 minutes ago Up 12 minutes 0.0.0.0:12400->5601/tcp, 0.0.0.0:12443->5643/tcp frc-cloud-uis-cloud-ui
60e5fa07dc86 docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.0.1 "/sbin/my_init" 12 minutes ago Up 12 minutes frc-curators-curator
9154e199a7d0 docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.0.1 "/sbin/my_init" 12 minutes ago Up 12 minutes 0.0.0.0:8764->8764/tcp, 0.0.0.0:12300->12300/tcp, 0.0.0.0:12343->12343/tcp frc-admin-consoles-admin-console
9f59225e61eb docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.0.1 "/sbin/my_init" 12 minutes ago Up 12 minutes frc-constructors-constructor
b1e6dc38839f docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.0.1 "/sbin/my_init" 12 minutes ago Up 12 minutes frc-services-forwarders-services-forwarder
d1720d4c7224 docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.0.1 "/sbin/my_init" 12 minutes ago Up 12 minutes 0.0.0.0:2112->2112/tcp frc-directors-director
d7a79ce55f45 docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.0.1 "/sbin/my_init" 12 minutes ago Up 12 minutes frc-allocators-allocator
ad29e93e5cf5 docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.0.1 "/sbin/my_init" 12 minutes ago Up 12 minutes frc-beats-runners-beats-runner
91b43147e6e2 docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.0.1 "/sbin/my_init" 12 minutes ago Up 12 minutes 0.0.0.0:9200->9200/tcp, 0.0.0.0:9243->9243/tcp, 0.0.0.0:9300->9300/tcp, 0.0.0.0:9343->9343/tcp frc-proxies-proxy
561baa3a1188 docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.0.1 "/sbin/my_init" 12 minutes ago Up 12 minutes frc-blueprints-blueprint
13415a142538 docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.0.1 "/sbin/my_init" 12 minutes ago Up 12 minutes frc-runners-runner
b379c65bf41f docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.0.1 "/sbin/my_init" 13 minutes ago Up 13 minutes 0.0.0.0:2191->2191/tcp, 0.0.0.0:12191->12191/tcp, 0.0.0.0:12898->12898/tcp, 0.0.0.0:13898->13898/tcp frc-zookeeper-servers-zookeeper
1f8429baf2d6 docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise:1.0.1 "/sbin/my_init" 13 minutes ago Up 13 minutes frc-client-forwarders-client-forwarder
62b91f01c1eb registry.adeo.com/docker-tools/docker-audit "/bin/sh -c /launch-d" 13 minutes ago Up 13 minutes docker-audit

8

@Aymeric

Can you run a docker images

My suspicion is that you have downloaded the cloud enterprise images but not the ES or Kibana images?

You should have something like:

docker.elastic.co/cloud-assets/elasticsearch          5.4.0-3             508c04049e0a        4 weeks ago         1.493 GB
docker.elastic.co/cloud-assets/elasticsearch          2.4.5-1             f6a89abfc31e        4 weeks ago         1.102 GB
docker.elastic.co/cloud-assets/kibana                 5.4.0-1             3ca2dc9d1495        6 weeks ago         722.4 MB
docker.elastic.co/cloud-assets/kibana                 4.6.4-0             ec1b760d33c5        5 months ago        1.063 GB

If the 5.4.x versions (I think 5.4.1 if you're running 1.0.1 - my images come from a 1.0.0 install) of those are missing then that is the problem, and you should just preload those.

If you do have those, then the place to look is

/mnt/data/elastic/<RUNNER-ID-OR-IP>/services/allocator/logs/allocator.log

You will likely see errors in there

9

Hi Alex,

Thanks for your reply

ES and Kibana images are loaded:
# docker images
REPOSITORY TAG IMAGE ID CREATED SIZE
registry.adeo.com/docker-tools/docker-audit latest ddc6a70f4c3c 15 hours ago 1.157 GB
docker.elastic.co/cloud-enterprise/elastic-cloud-enterprise 1.0.1 75fd506d6f00 10 days ago 548.2 MB
docker.elastic.co/cloud-enterprise/kibana 5.3.0-0 1a4c78a54e62 11 weeks ago 988.3 MB
docker.elastic.co/cloud-enterprise/elasticsearch 5.3.0-1 f8efb7114256 12 weeks ago 1.21 GB
docker.elastic.co/cloud-enterprise/elasticsearch 2.4.4-1 f19d395f6016 4 months ago 914.5 MB
docker.elastic.co/cloud-enterprise/kibana 4.6.4 72254a092199 4 months ago 1.063 GB

Here's the content of allocator.log file (will put it in 2 posts as very verbose :slight_smile: ):

[2017-06-16 14:41:51,652][ERROR][no.found.runner.allocation.elasticsearch.ElasticsearchDockerContainerManager] Unexpected error during allocation {"ec_container_kind":"elasticsearch","ec_container_group":"96ca1e9679e845518e2b87727001ce93","ec_container_name":"instance-0000000000"}
no.found.docker.DockerApiException: Unable to pull image [docker.elastic.co/cloud-assets/elasticsearch:5.4.1-2]
Docker API request: [HttpRequest(POST,http://localhost:2375/v1.19/images/create?fromImage=docker.elastic.co/cloud-assets/elasticsearch:5.4.1-2,List(),Empty,HTTP/1.1)], response: [HttpResponse(500 Internal Server Error,HttpEntity(text/plain; charset=UTF-8,Get https://docker.elastic.co/v1/_ping: dial tcp 34.194.95.255:443: i/o timeout
),List(Content-Type: text/plain; charset=UTF-8, Server: Docker/1.12.6 (linux), X-Content-Type-Options: nosniff, Date: Fri, 16 Jun 2017 14:41:51 GMT, Content-Length: 80),HTTP/1.1)]
at no.found.docker.DockerContainerManager$$anonfun$pullImage$2.apply(DockerContainerManager.scala:385)
at no.found.docker.DockerContainerManager$$anonfun$pullImage$2.apply(DockerContainerManager.scala:381)
at scala.util.Success$$anonfun$map$1.apply(Try.scala:237)
at scala.util.Try$.apply(Try.scala:192)
at scala.util.Success.map(Try.scala:237)
at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:237)
at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:237)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at no.found.concurrent.WrappedRunnable.run(ControllableExecutionContextWrapper.scala:55)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)
[2017-06-16 14:41:51,653][ERROR][akka.dispatch.Dispatcher ] Unable to pull image [docker.elastic.co/cloud-assets/elasticsearch:5.4.1-2]
Docker API request: [HttpRequest(POST,http://localhost:2375/v1.19/images/create?fromImage=docker.elastic.co/cloud-assets/elasticsearch:5.4.1-2,List(),Empty,HTTP/1.1)], response: [HttpResponse(500 Internal Server Error,HttpEntity(text/plain; charset=UTF-8,Get https://docker.elastic.co/v1/_ping: dial tcp 34.194.95.255:443: i/o timeout
),List(Content-Type: text/plain; charset=UTF-8, Server: Docker/1.12.6 (linux), X-Content-Type-Options: nosniff, Date: Fri, 16 Jun 2017 14:41:51 GMT, Content-Length: 80),HTTP/1.1)] {}
no.found.docker.DockerApiException: Unable to pull image [docker.elastic.co/cloud-assets/elasticsearch:5.4.1-2]
Docker API request: [HttpRequest(POST,http://localhost:2375/v1.19/images/create?fromImage=docker.elastic.co/cloud-assets/elasticsearch:5.4.1-2,List(),Empty,HTTP/1.1)], response: [HttpResponse(500 Internal Server Error,HttpEntity(text/plain; charset=UTF-8,Get https://docker.elastic.co/v1/_ping: dial tcp 34.194.95.255:443: i/o timeout
),List(Content-Type: text/plain; charset=UTF-8, Server: Docker/1.12.6 (linux), X-Content-Type-Options: nosniff, Date: Fri, 16 Jun 2017 14:41:51 GMT, Content-Length: 80),HTTP/1.1)]
at no.found.docker.DockerContainerManager$$anonfun$pullImage$2.apply(DockerContainerManager.scala:385)
at no.found.docker.DockerContainerManager$$anonfun$pullImage$2.apply(DockerContainerManager.scala:381)
at scala.util.Success$$anonfun$map$1.apply(Try.scala:237)
at scala.util.Try$.apply(Try.scala:192)
at scala.util.Success.map(Try.scala:237)
at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:237)
at scala.concurrent.Future$$anonfun$map$1.apply(Future.scala:237)
at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:32)
at no.found.concurrent.WrappedRunnable.run(ControllableExecutionContextWrapper.scala:55)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:40)
at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:397)
at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

10

[2017-06-16 14:41:51,732][INFO ][no.found.runner.managers.XFSQuotaManager] Adding quota using project id [XFSProjectName(es,96ca1e9679e845518e2b87727001ce93,instance-0000000000)] with hard limit: [106496] MB {"ec_container_kind":"elasticsearch","ec_container_group":"96ca1e9679e845518e2b87727001ce93","ec_container_name":"instance-0000000000"}
[2017-06-16 14:41:51,739][WARN ][no.found.runner.managers.XFSQuotaManager] Command [[sudo, -n, xfs_quota, -x, -c, project -s -p /data/elastic/10.11.176.72/services/allocator/containers/elasticsearch/96ca1e9679e845518e2b87727001ce93/instance-0000000000/data 12769, /data]] returned status code [0]] with output: [E: [xfs_quota: cannot setup path for mount /data: No such device or address]] {}
[2017-06-16 14:41:51,744][WARN ][no.found.runner.managers.XFSQuotaManager] Command [[sudo, -n, xfs_quota, -x, -c, project -s -p /data/elastic/10.11.176.72/services/allocator/containers/elasticsearch/96ca1e9679e845518e2b87727001ce93/instance-0000000000/logs 12769, /data]] returned status code [0]] with output: [E: [xfs_quota: cannot setup path for mount /data: No such device or address], E: [xfs_quota: cannot setup path for mount /data: No such device or address]] {}
[2017-06-16 14:41:51,751][WARN ][no.found.runner.managers.XFSQuotaManager] Command [[sudo, -n, xfs_quota, -x, -c, limit -p bhard=106496m rtbhard=106496m 12769, /data]] returned status code [0]] with output: [E: [xfs_quota: cannot setup path for mount /data: No such device or address], E: [xfs_quota: cannot setup path for mount /data: No such device or address], E: [xfs_quota: cannot setup path for mount /data: No such device or address]] {}
[2017-06-16 14:41:51,775][INFO ][no.found.runner.docker.DockerAllocator] Creating container {"ec_container_kind":"elasticsearch","ec_container_group":"96ca1e9679e845518e2b87727001ce93","ec_container_name":"instance-0000000000"}
[2017-06-16 14:41:51,776][INFO ][no.found.runner.allocation.elasticsearch.ElasticsearchDockerContainer] Creating container from [ElasticsearchDockerContainer(ElasticsearchContainerId(96ca1e9679e845518e2b87727001ce93,instance-0000000000),-,List(),{"data":{"name":"admin-console-elasticsearch"},"kind":"elasticsearch","resources":{"cpu":{"hard_limit":false}},"shield":{"system_key":"-","found_users":{"ece-admin-cluster-indexer":{"roles":["found-internal-admin"],"username":"ece-admin-cluster-indexer","password_hash":"-"},"found-internal-admin-proxy":{"roles":["found-internal-admin"],"username":"found-internal-admin-proxy","password_hash":"-","valid_until":"2059-06-16T14:06:25.764Z"},"found-internal-constructor":{"roles":["found-internal-admin","superuser"],"username":"found-internal-constructor","password":"-","password_hash":"-","valid_until":"2022-06-16T14:06:41.893Z"},"found-local-allocator":{"password_hash":"-","roles":["found-internal-admin"]}}},"snapshot":{"enabled":false,"suspended":{"constructor":true}},"user_id":"root","system_owned":true,"source":{"facilitator":"bootstrap","action":"create-admin-console-backend","date":"2017-06-16T14:06:25.754Z"}},{"instance_capacity":4096,"ssd":false,"elasticsearch":{"version":"5.4.1","confirmed_election":false,"plugins":["repository-s3","found-elasticsearch","x-pack"],"user_bundles":[],"user_plugins":[],"user_settings":{},"user_settings_overrides":{},"gateway_type":"local","seed_hosts":[]},"logical_zone_name":"zone-0","start":true,"quorum_size":1,"maintenance":true,"source":{"facilitator":"constructor","action":"elasticsearch.create-instances","date":"2017-06-16T14:06:42.323Z"},"allocator_name":"10.11.176.72","allocator_id":"10.11.176.72","allocator_zone_name":"ece-region-1a"},{"forward_auth_headers":true,"shield":{"allow_anonymous":false,"esusers":{"roles":{},"users":{},"users_roles":{}}},"source":{"facilitator":"bootstrap","action":"create-admin-console-backend","date":"2017-06-16T14:06:25.754Z"}},no.found.template.ZooKeeperResourceLocator@50125e20,DefaultAllocatorAllocationEnvironment(13065,10.11.176.72,ece-region-1a,10.11.176.72,found/{kind}:{version},found/{kind}:{version},26,4,/data/elastic/10.11.176.72/services/allocator/containers,/app/state,state/runner-initial-containers-content.json,no.found.zookeeper.VersionCache@687c39ba,no.found.runner.AllocatorBootstrap$$anon$1@79d8d26e,no.found.runner.managers.ContainerIdsManager@30ec3914,no.found.runner.managers.XFSQuotaSupport@724d9277,no.found.template.VersionedTemplateManager@7b75d46d,no.found.runner.managers.ZooKeeperManager@448132cf,no.found.runner.managers.NetworkResourcesManager@157497d2,no.found.runner.settings.AllocatorSettings@b346121))]: [{"Image":"docker.elastic.co/cloud-assets/elasticsearch:5.4.1-2","Labels":{"co.elastic.cloud.allocator.id":"10.11.176.72","co.elastic.cloud.allocator.zone":"ece-region-1a","co.elastic.cloud.allocator.instance_id":"instance-0000000000","co.elastic.cloud.allocator.cluster_id":"96ca1e9679e845518e2b87727001ce93","co.elastic.cloud.allocator.user_id":"root","co.elastic.cloud.allocator.kind":"elasticsearch","co.elastic.cloud.allocator.type":"elasticsearch","co.elastic.cloud.allocator.type_version":"5.4.1"},"ExposedPorts":{"18127/tcp":{},"19682/tcp":{}},"Volumes":{"/app":{}},"HostConfig":{"CpuShares":2,"Memory":4294967296,"MemorySwap":4294967296,"BlkioWeight":10,"CpuPeriod":100000,"Binds":["/data/elastic/10.11.176.72/services/allocator/containers/elasticsearch/96ca1e9679e845518e2b87727001ce93/instance-0000000000:/app"],"RestartPolicy":{"Name":"always"},"ExtraHosts":["containerhost:${ALLOCATOR_HOST_IP}","dockerhost:${ALLOCATOR_HOST_IP}"],"PortBindings":{"18127/tcp":[{"HostIp":"0.0.0.0","HostPort":"18127"}],"19682/tcp":[{"HostIp":"0.0.0.0","HostPort":"19682"}]},"CapAdd":["SETUID","SETGID","CHOWN","DAC_OVERRIDE"],"CapDrop":["ALL"]},"Env":["ELASTIC_UID=1001","ELASTIC_GID=1001"]}] {"ec_container_kind":"elasticsearch","ec_container_group":"96ca1e9679e845518e2b87727001ce93","ec_container_name":"instance-0000000000"}
[2017-06-16 14:41:51,787][INFO ][no.found.docker.DockerContainerManager] Creating container [fac-96ca1e9679e845518e2b87727001ce93-instance-0000000000] {"ec_container_kind":"elasticsearch","ec_container_group":"96ca1e9679e845518e2b87727001ce93","ec_container_name":"instance-0000000000"}
[2017-06-16 14:41:51,792][WARN ][no.found.docker.DockerContainerManager] Unable to find auth for registry [docker.elastic.co] in auths. Have [List()], trying without auth. {}
[2017-06-16 14:41:51,792][INFO ][no.found.docker.DockerContainerManager] Pulling image [docker.elastic.co/cloud-assets/elasticsearch:5.4.1-2] {}
[2017-06-16 14:41:56,748][INFO ][no.found.runner.docker.ContainerInspector] Container not found. {"ec_container_kind":"elasticsearch","ec_container_group":"96ca1e9679e845518e2b87727001ce93","ec_container_name":"instance-0000000000"}

11

@Aymeric

Perfect! That solves the problem:

Unable to pull image [docker.elastic.co/cloud-assets/elasticsearch:5.4.1-2

vs

docker.elastic.co/cloud-assets/elasticsearch 5.3.0-1

So you have old ES and Kibana images (we'll need to look into whether the docs are out-of-date, obviously we need a better way of reporting what the current images offline installs should be using, we'll work on that, EDIT: docs are out of date https://www.elastic.co/guide/en/cloud-enterprise/current/ece-installing-offline.html I created an issue to update them)

So if you pull docker.elastic.co/cloud-assets/elasticsearch:5.4.1-2 and docker.elastic.co/cloud-assets/kibana:5.4.1-2and then re-install, it should work

Alex

12

(Post install, if you want to test 2.x you should also pull docker.elastic.co/cloud-assets/elasticsearch:2.4.5-1 and docker.elastic.co/cloud-assets/kibana:4.6.4-0)

13

Great, many thanks for your help. Right: as a newbie, I stick to the documentation :slight_smile:
I'll pull the latest images, reinstall and let you know asap.

Thx!
Aymeric

14

This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.