# Elasticsearch REST API layer is not available. Forcefully terminated cluster **URL:** \ **Category:** Elasticsearch\ **Tags:** rally\ **Created:** [November 25, 2019, 7:59am UTC](https://discuss.elastic.co/t/elasticsearch-rest-api-layer-is-not-available-forcefully-terminated-cluster/209239 "2019-11-25T07:59:10Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![vishal\_viswanathan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vishal_viswanathan/32/58251_2.png) [@vishal\_viswanathan](https://discuss.elastic.co/u/vishal_viswanathan)\ **Post date:** [November 25, 2019, 7:59am UTC](https://discuss.elastic.co/t/elasticsearch-rest-api-layer-is-not-available-forcefully-terminated-cluster/209239/1 "2019-11-25T07:59:10Z")
Hello Getting the below error while running rally If they are absent, running this track will fail or lead to unexpected results. [ERROR] Cannot race. Could not launch cluster Elasticsearch REST API layer is not available. Forcefully terminated cluster. > Edit from moderator > Additional logs: ```auto File "/usr/local/lib/python3.7/site-packages/esrally/racecontrol.py", line 366, in run pipeline(cfg) File "/usr/local/lib/python3.7/site-packages/esrally/racecontrol.py", line 78, in call self.target(cfg) File "/usr/local/lib/python3.7/site-packages/esrally/racecontrol.py", line 302, in benchmark_only return race(cfg, external=True) File "/usr/local/lib/python3.7/site-packages/esrally/racecontrol.py", line 254, in race raise exceptions.RallyError(result.message, result.cause) esrally.exceptions.RallyError: ('Could not launch cluster', Elasticsearch REST API layer is not available. Forcefully terminated cluster.) ``` ---
**Author:** ![dliappis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dliappis/32/56174_2.png) [@dliappis](https://discuss.elastic.co/u/dliappis)\ **Post date:** [November 25, 2019, 11:25am UTC](https://discuss.elastic.co/t/elasticsearch-rest-api-layer-is-not-available-forcefully-terminated-cluster/209239/2 "2019-11-25T11:25:23Z")
Hello, This question looks identical to the previous one you opened: [Rally cannot start](https://discuss.elastic.co/t/rally-cannot-start/209233). I've now closed [Rally cannot start](https://discuss.elastic.co/t/rally-cannot-start/209233) and will update the description here with the additional logs from the other topic. Dimitris ---
**Author:** ![dliappis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dliappis/32/56174_2.png) [@dliappis](https://discuss.elastic.co/u/dliappis)\ **Post date:** [November 25, 2019, 2:02pm UTC](https://discuss.elastic.co/t/elasticsearch-rest-api-layer-is-not-available-forcefully-terminated-cluster/209239/3 "2019-11-25T14:02:56Z")
@vishal_viswanathan to your question, what is your Rally command and are you trying to benchmark an existing cluster or have Rally set it up for you? If the former you should take a look a the [benchmark-only](https://esrally.readthedocs.io/en/stable/pipelines.html#benchmark-only) pipeline. ---
**Author:** ![vishal\_viswanathan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vishal_viswanathan/32/58251_2.png) [@vishal\_viswanathan](https://discuss.elastic.co/u/vishal_viswanathan)\ **Post date:** [December 2, 2019, 3:55am UTC](https://discuss.elastic.co/t/elasticsearch-rest-api-layer-is-not-available-forcefully-terminated-cluster/209239/4 "2019-12-02T03:55:27Z")
@dliappis this is the error I am getting rally@d7733066d34e:~$ esrally --track=pmc --target-hosts=IP:9300,IP:9300,IP:9300 --pipeline=benchmark-only --distribution-version=6.8.2 ``` ________ ``` / \_\_ \_\_\_\_ _/ / /_ \_\_ / /_/ / \_\_ `/ / / / / / / , / // / / / // / /_/ |_|\ __,_/_/_/\__ , / /\_\_\_\_/ [WARNING] No Internet connection detected. Automatic download of track data sets etc. is disabled. * * * \*\*\*\*\*\*\*\*\*\*\*\*\*\* WARNING: A dark dungeon lies ahead of you \*\*\*\*\*\*\*\*\*\*\*\*\*\* * * * Rally does not have control over the configuration of the benchmarked Elasticsearch cluster. Be aware that results may be misleading due to problems with the setup. Rally is also not able to gather lots of metrics at all (like CPU usage of the benchmarked cluster) or may even produce misleading metrics (like the index size). * * * \*\*\*\*\*\* Use this pipeline only if you are aware of the tradeoffs. \*\*\*\*\*\* \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* Watch your step! \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* * * * [WARNING] Ensure that these settings are defined in elasticsearch.yml: { "search.default\_search\_timeout": "-1" } If they are absent, running this track will fail or lead to unexpected results. [ERROR] Cannot race. Could not launch cluster Elasticsearch REST API layer is not available. Forcefully terminated cluster. Getting further help: * * * - Check the log files in /rally/.rally/logs for errors. - Read the documentation at [https://esrally.readthedocs.io/en/1.3.0/](https://esrally.readthedocs.io/en/1.3.0/) - Ask a question on the forum at [https://discuss.elastic.co/c/elasticsearch/rally](https://discuss.elastic.co/c/elasticsearch/rally) - Raise an issue at [https://github.com/elastic/rally/issues](https://github.com/elastic/rally/issues) and include the log files in /rally/.rally/logs. * * * [INFO] FAILURE (took 58 seconds) can you take a look and provide a valuable solution? ---
**Author:** ![dliappis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dliappis/32/56174_2.png) [@dliappis](https://discuss.elastic.co/u/dliappis)\ **Post date:** [December 2, 2019, 10:14am UTC](https://discuss.elastic.co/t/elasticsearch-rest-api-layer-is-not-available-forcefully-terminated-cluster/209239/5 "2019-12-02T10:14:36Z")
Hi, You shouldn't be specifying [--distribution-version](https://esrally.readthedocs.io/en/stable/command_line_reference.html#distribution-version) in conjunction with `--pipeline=benchmark-only`; you are benchmarking an external cluster which has been provisioned already so it has whatever version it has (and Rally will discover the version and store it in the metric store if configured for future reference). I raised an issue to make this clearer, at the least through the docs: [https://github.com/elastic/rally/issues/833](https://github.com/elastic/rally/issues/833) Rgs, Dimitris ---
**Author:** ![dliappis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dliappis/32/56174_2.png) [@dliappis](https://discuss.elastic.co/u/dliappis)\ **Post date:** [December 2, 2019, 11:28am UTC](https://discuss.elastic.co/t/elasticsearch-rest-api-layer-is-not-available-forcefully-terminated-cluster/209239/6 "2019-12-02T11:28:13Z")
> [@vishal\_viswanathan](#): > > --target-hosts=IP:9300,IP:9300,IP:9300 Also note you are using port `9300` which, by default, is the transport port; the port you should be using for Elasticsearch is the http port instead, typically `9200`. ---
**Author:** ![vishal\_viswanathan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vishal_viswanathan/32/58251_2.png) [@vishal\_viswanathan](https://discuss.elastic.co/u/vishal_viswanathan)\ **Post date:** [December 3, 2019, 12:03pm UTC](https://discuss.elastic.co/t/elasticsearch-rest-api-layer-is-not-available-forcefully-terminated-cluster/209239/7 "2019-12-03T12:03:35Z")
@dliappis Getting below rally@d7733066d34e:~$ esrally --track=pmc --target-hosts=IP:9200,IP:9200,IP:9200 --pipeline=benchmark-only ``` ________ ``` / \_\_ \_\_\_\_ _/ / /_ \_\_ / /_/ / \_\_ `/ / / / / / / , / // / / / // / /_/ |_|\ __,_/_/_/\__ , / /\_\_\_\_/ [WARNING] No Internet connection detected. Automatic download of track data sets etc. is disabled. * * * \*\*\*\*\*\*\*\*\*\*\*\*\*\* WARNING: A dark dungeon lies ahead of you \*\*\*\*\*\*\*\*\*\*\*\*\*\* * * * Rally does not have control over the configuration of the benchmarked Elasticsearch cluster. Be aware that results may be misleading due to problems with the setup. Rally is also not able to gather lots of metrics at all (like CPU usage of the benchmarked cluster) or may even produce misleading metrics (like the index size). * * * \*\*\*\*\*\* Use this pipeline only if you are aware of the tradeoffs. \*\*\*\*\*\* \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* Watch your step! \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* * * * [ERROR] Cannot race. Error in race control (('A distribution version is required. Please specify it with --distribution-version.', None)) Getting further help: * * * - Check the log files in /rally/.rally/logs for errors. - Read the documentation at [https://esrally.readthedocs.io/en/1.3.0/](https://esrally.readthedocs.io/en/1.3.0/) - Ask a question on the forum at [https://discuss.elastic.co/c/elasticsearch/rally](https://discuss.elastic.co/c/elasticsearch/rally) - Raise an issue at [https://github.com/elastic/rally/issues](https://github.com/elastic/rally/issues) and include the log files in /rally/.rally/logs. * * * [INFO] FAILURE (took 29 seconds) ---
**Author:** ![dliappis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dliappis/32/56174_2.png) [@dliappis](https://discuss.elastic.co/u/dliappis)\ **Post date:** [December 3, 2019, 1:33pm UTC](https://discuss.elastic.co/t/elasticsearch-rest-api-layer-is-not-available-forcefully-terminated-cluster/209239/8 "2019-12-03T13:33:54Z")
Hi, 1. What's the Rally version? `esrally --version` 2. Can you paste the most recent Rally logs (e.g. `tail -100 ~/.rally/logs/rally.log`) 3. Can you confirm you can reach the Elasticsearch HTTP API from the Rally node? e.g. try `curl http://IP:9200` ---
**Author:** ![vishal\_viswanathan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vishal_viswanathan/32/58251_2.png) [@vishal\_viswanathan](https://discuss.elastic.co/u/vishal_viswanathan)\ **Post date:** [December 5, 2019, 10:18am UTC](https://discuss.elastic.co/t/elasticsearch-rest-api-layer-is-not-available-forcefully-terminated-cluster/209239/9 "2019-12-05T10:18:16Z")
Hi, 1. rally@d7733066d34e:~$ esrally --version esrally 1.3.0 2. rally@d7733066d34e:~$ tail -100 ~/.rally/logs/rally.log File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 333, in increment raise six.reraise(type(error), error, \_stacktrace) File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 686, in reraise raise value File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 601, in urlopen chunked=chunked) File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 357, in \_make\_request conn.request(method, url, \*\*httplib\_request\_kw) File "/usr/local/lib/python3.7/http/client.py", line 1229, in request self.\_send\_request(method, url, body, headers, encode\_chunked) File "/usr/local/lib/python3.7/http/client.py", line 1275, in \_send\_request self.endheaders(body, encode\_chunked=encode\_chunked) File "/usr/local/lib/python3.7/http/client.py", line 1224, in endheaders self.\_send\_output(message\_body, encode\_chunked=encode\_chunked) File "/usr/local/lib/python3.7/http/client.py", line 1016, in \_send\_output self.send(msg) File "/usr/local/lib/python3.7/http/client.py", line 956, in send self.connect() File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 166, in connect conn = self.\_new\_conn() File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 150, in \_new\_conn self, "Failed to establish a new connection: %s" % e) urllib3.exceptions.NewConnectionError: \: Failed to establish a new connection: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/esrally/mechanic/mechanic.py", line 218, in cluster\_distribution\_version return es.info()["version"]["number"] File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/utils.py", line 76, in \_wrapped return func(\*args, params=params, \*\*kwargs) File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/ **init**.py", line 241, in info return self.transport.perform\_request('GET', '/', params=params) File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 314, in perform\_request status, headers\_response, data = connection.perform\_request(method, url, params, body, headers=headers, ignore=ignore, timeout=timeout) File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http\_urllib3.py", line 175, in perform\_request raise ConnectionError('N/A', str(e), e) elasticsearch.exceptions.ConnectionError: ConnectionError(\: Failed to establish a new connection: [Errno 113] No route to host) caused by: NewConnectionError(\: Failed to establish a new connection: [Errno 113] No route to host) 2019-12-03 12:01:19,965 ActorAddr-(T|:44702)/PID:513 esrally.actor ERROR Error in race control Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/esrally/actor.py", line 84, in guard return f(self, msg, sender) File "/usr/local/lib/python3.7/site-packages/esrally/racecontrol.py", line 119, in receiveMsg\_Setup self.setup(msg, sender) File "/usr/local/lib/python3.7/site-packages/esrally/racecontrol.py", line 200, in setup raise exceptions.SystemSetupError("A distribution version is required. Please specify it with --distribution-version.") esrally.exceptions.SystemSetupError: ('A distribution version is required. Please specify it with --distribution-version.', None) 2019-12-03 12:01:19,970 ActorAddr-(T|:44702)/PID:513 esrally.actor INFO BenchmarkActor received unknown message [ActorExitRequest] (ignoring). 2019-12-03 12:01:22,974 -not-actor-/PID:506 esrally.rally INFO Attempting to shutdown internal actor system. 2019-12-03 12:01:22,977 -not-actor-/PID:512 root INFO ActorSystem Logging Shutdown 2019-12-03 12:01:22,998 -not-actor-/PID:511 root INFO ---- Actor System shutdown 2019-12-03 12:01:22,999 -not-actor-/PID:506 esrally.rally INFO Actor system is still running. Waiting... 2019-12-03 12:01:24,0 -not-actor-/PID:506 esrally.rally INFO Shutdown completed. 2019-12-03 12:01:24,0 -not-actor-/PID:506 esrally.rally ERROR Cannot run subcommand [race]. Traceback (most recent call last): File "/usr/local/lib/python3.7/site-packages/esrally/rally.py", line 515, in dispatch\_sub\_command race(cfg) File "/usr/local/lib/python3.7/site-packages/esrally/rally.py", line 433, in race with\_actor\_system(lambda c: racecontrol.run(c), cfg) File "/usr/local/lib/python3.7/site-packages/esrally/rally.py", line 461, in with\_actor\_system runnable(cfg) File "/usr/local/lib/python3.7/site-packages/esrally/rally.py", line 433, in with\_actor\_system(lambda c: racecontrol.run(c), cfg) File "/usr/local/lib/python3.7/site-packages/esrally/racecontrol.py", line 369, in run raise e File "/usr/local/lib/python3.7/site-packages/esrally/racecontrol.py", line 366, in run pipeline(cfg) File "/usr/local/lib/python3.7/site-packages/esrally/racecontrol.py", line 78, in **call** self.target(cfg) File "/usr/local/lib/python3.7/site-packages/esrally/racecontrol.py", line 302, in benchmark\_only return race(cfg, external=True) File "/usr/local/lib/python3.7/site-packages/esrally/racecontrol.py", line 254, in race raise exceptions.RallyError(result.message, result.cause) esrally.exceptions.RallyError: ("Error in race control (('A distribution version is required. Please specify it with --distribution-version.', None))", None) 3) I have installed the docker ally container on one of the ES node. There is no seperate node. ---
**Author:** ![dliappis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dliappis/32/56174_2.png) [@dliappis](https://discuss.elastic.co/u/dliappis)\ **Post date:** [December 5, 2019, 10:37am UTC](https://discuss.elastic.co/t/elasticsearch-rest-api-layer-is-not-available-forcefully-terminated-cluster/209239/10 "2019-12-05T10:37:27Z")
So logs tell us that Rally is unable to connect to the provided host:port list: > elasticsearch.exceptions.ConnectionError: ConnectionError(\: Failed to establish a new connection: [Errno 113] No route to host) caused by: NewConnectionError(\: Failed to establish a new connection: [Errno 113] No route to host) I suggest you confirm connectivity from the Rally host as indicated in step 3. in the [last comment](https://discuss.elastic.co/t/elasticsearch-rest-api-layer-is-not-available-forcefully-terminated-cluster/209239/8). ---
**Author:** ![vishal\_viswanathan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vishal_viswanathan/32/58251_2.png) [@vishal\_viswanathan](https://discuss.elastic.co/u/vishal_viswanathan)\ **Post date:** [December 9, 2019, 8:54am UTC](https://discuss.elastic.co/t/elasticsearch-rest-api-layer-is-not-available-forcefully-terminated-cluster/209239/11 "2019-12-09T08:54:08Z")
@dliappis So for running rally as docker container we need to open a port? ---
**Author:** ![dliappis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dliappis/32/56174_2.png) [@dliappis](https://discuss.elastic.co/u/dliappis)\ **Post date:** [December 9, 2019, 11:02am UTC](https://discuss.elastic.co/t/elasticsearch-rest-api-layer-is-not-available-forcefully-terminated-cluster/209239/12 "2019-12-09T11:02:35Z")
You need to confirm that wherever Rally runs on (i.e. inside a real OS, VM or a container) you are able to connect to the IP:port of the Elasticsearch nodes. ---
**Author:** ![vishal\_viswanathan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vishal_viswanathan/32/58251_2.png) [@vishal\_viswanathan](https://discuss.elastic.co/u/vishal_viswanathan)\ **Post date:** [December 9, 2019, 12:08pm UTC](https://discuss.elastic.co/t/elasticsearch-rest-api-layer-is-not-available-forcefully-terminated-cluster/209239/13 "2019-12-09T12:08:16Z")
I am running rally inside the docker container. I don't have any ping or nc command to check the connectivity to ES nodes. Any suggestions? ---
**Author:** ![vishal\_viswanathan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vishal_viswanathan/32/58251_2.png) [@vishal\_viswanathan](https://discuss.elastic.co/u/vishal_viswanathan)\ **Post date:** [December 9, 2019, 12:36pm UTC](https://discuss.elastic.co/t/elasticsearch-rest-api-layer-is-not-available-forcefully-terminated-cluster/209239/14 "2019-12-09T12:36:27Z")
Here are results From rally docker conatiner, I am able to connect to three test node. rally@d7733066d34e:~ curl -k -u elastic:\*\*\*\*\*\*\* "https://\*.\*.\*.\*:9200" { "name" : "c4\_testelk\_node\_02", "cluster\_name" : "c4\_testelk", "cluster\_uuid" : "Lw94O89STI6OoFHJEcbGTw", "version" : { "number" : "6.8.2", "build\_flavor" : "default", "build\_type" : "docker", "build\_hash" : "b506955", "build\_date" : "2019-07-24T15:24:41.545295Z", "build\_snapshot" : false, "lucene\_version" : "7.7.0", "minimum\_wire\_compatibility\_version" : "5.6.0", "minimum\_index\_compatibility\_version" : "5.0.0" }, "tagline" : "You Know, for Search" } rally@d7733066d34e:~ curl -k -u elastic:\*\*\*\*\*\*\* "https://_._._._:9200" { "name" : "c4\_testelk\_node\_03", "cluster\_name" : "c4\_testelk", "cluster\_uuid" : "Lw94O89STI6OoFHJEcbGTw", "version" : { "number" : "6.8.2", "build\_flavor" : "default", "build\_type" : "docker", "build\_hash" : "b506955", "build\_date" : "2019-07-24T15:24:41.545295Z", "build\_snapshot" : false, "lucene\_version" : "7.7.0", "minimum\_wire\_compatibility\_version" : "5.6.0", "minimum\_index\_compatibility\_version" : "5.0.0" }, "tagline" : "You Know, for Search" } rally@d7733066d34e:~$ curl -k -u elastic:\*\*\*\*\*\*\* "https://_._._._:9200" { "name" : "c4\_testelk\_node\_01", "cluster\_name" : "c4\_testelk", "cluster\_uuid" : "Lw94O89STI6OoFHJEcbGTw", "version" : { "number" : "6.8.2", "build\_flavor" : "default", "build\_type" : "docker", "build\_hash" : "b506955", "build\_date" : "2019-07-24T15:24:41.545295Z", "build\_snapshot" : false, "lucene\_version" : "7.7.0", "minimum\_wire\_compatibility\_version" : "5.6.0", "minimum\_index\_compatibility\_version" : "5.0.0" }, "tagline" : "You Know, for Search" } ---
**Author:** ![dliappis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dliappis/32/56174_2.png) [@dliappis](https://discuss.elastic.co/u/dliappis)\ **Post date:** [December 9, 2019, 3:36pm UTC](https://discuss.elastic.co/t/elasticsearch-rest-api-layer-is-not-available-forcefully-terminated-cluster/209239/15 "2019-12-09T15:36:21Z")
So from your last reply I see you are accessing the Elasticsearch cluster with https including authentication. Your [earlier](https://discuss.elastic.co/t/elasticsearch-rest-api-layer-is-not-available-forcefully-terminated-cluster/209239/7) reply to how you run Rally only specified `target-hosts`. Please take a look in the [client-options](https://esrally.readthedocs.io/en/stable/command_line_reference.html#client-options) documentation page to see how to specify credentials, use TLS and not verify certificates (which is what you are doing when you specify `-k` in curl). ---
**Author:** ![vishal\_viswanathan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vishal_viswanathan/32/58251_2.png) [@vishal\_viswanathan](https://discuss.elastic.co/u/vishal_viswanathan)\ **Post date:** [December 10, 2019, 11:53am UTC](https://discuss.elastic.co/t/elasticsearch-rest-api-layer-is-not-available-forcefully-terminated-cluster/209239/16 "2019-12-10T11:53:13Z")
After running with the below authentication, I am getting below error rally@d7733066d34e:~$ esrally --track=pmc --target-hosts=IP:9200,IP:9200,IP:9200 --pipeline=benchmark-only --client-options="use\_ssl:true,verify\_certs:false,basic\_auth\_user:'elastic',basic\_auth\_password:'\*\*\*\*\*\*'" ``` ________ ``` / \_\_ \_\_\_\_ _/ / /_ \_\_ / /_/ / \_\_ `/ / / / / / / , / // / / / // / /_/ |_|\ __,_/_/_/\__ , / /\_\_\_\_/ [INFO] You did not provide an explicit timeout in the client options. Assuming default of 10 seconds. [WARNING] No Internet connection detected. Automatic download of track data sets etc. is disabled. * * * \*\*\*\*\*\*\*\*\*\*\*\*\*\* WARNING: A dark dungeon lies ahead of you \*\*\*\*\*\*\*\*\*\*\*\*\*\* * * * Rally does not have control over the configuration of the benchmarked Elasticsearch cluster. Be aware that results may be misleading due to problems with the setup. Rally is also not able to gather lots of metrics at all (like CPU usage of the benchmarked cluster) or may even produce misleading metrics (like the index size). * * * \*\*\*\*\*\* Use this pipeline only if you are aware of the tradeoffs. \*\*\*\*\*\* \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* Watch your step! \*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\* * * * [WARNING] Ensure that these settings are defined in elasticsearch.yml: { "search.default\_search\_timeout": "-1" } If they are absent, running this track will fail or lead to unexpected results. [INFO] Racing on track [pmc], challenge [append-no-conflicts] and car ['external'] with version [6.8.2]. [WARNING] indexing\_total\_time is 1266 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading. [WARNING] refresh\_total\_time is 3787 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading. [WARNING] flush\_total\_time is 10141 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading. [ERROR] Cannot race. Error in track preparator (('Cannot find /rally/.rally/benchmarks/data/pmc/documents.json.bz2. Please disable offline mode and retry again.', None)) I cannot run the pipeline bench in offline mode as the docker images from rally doesn't allow as it will only run in pipleine-only mode. Also I am not able to find the below folder inside docker /rally/.rally/benchmarks/data/pmc/documents.json.bz2. ---
**Author:** ![dliappis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dliappis/32/56174_2.png) [@dliappis](https://discuss.elastic.co/u/dliappis)\ **Post date:** [December 11, 2019, 3:43pm UTC](https://discuss.elastic.co/t/elasticsearch-rest-api-layer-is-not-available-forcefully-terminated-cluster/209239/17 "2019-12-11T15:43:24Z")
Hi, Rally needs to download the necessary tracks (in your case `pmc`) and it appears it doesn't have access to the internet (from within your container at least) based on the error message: `[WARNING] No Internet connection detected. Automatic download of track data sets etc. is disabled.` Please take a look at the [Offline usage section](https://esrally.readthedocs.io/en/stable/offline.html#offline-usage) where you will instructions on downloading the tracks yourself (e.g. using this [offline script](https://raw.githubusercontent.com/elastic/rally-tracks/master/download.sh)) and then ensure you are bind mounting them in the Docker container as described [in the docs](https://esrally.readthedocs.io/en/stable/docker.html#persistence). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 8, 2020, 3:43pm UTC](https://discuss.elastic.co/t/elasticsearch-rest-api-layer-is-not-available-forcefully-terminated-cluster/209239/18 "2020-01-08T15:43:48Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.