Elasticsearch REST API layer is not available. Forcefully terminated cluster

Elastic Stack Elasticsearch
1

Hello Getting the below error while running rally

If they are absent, running this track will fail or lead to unexpected results.
[ERROR] Cannot race. Could not launch cluster
Elasticsearch REST API layer is not available. Forcefully terminated cluster.

Edit from moderator
Additional logs:

File "/usr/local/lib/python3.7/site-packages/esrally/racecontrol.py", line 366, in run
pipeline(cfg)
File "/usr/local/lib/python3.7/site-packages/esrally/racecontrol.py", line 78, in call
self.target(cfg)
File "/usr/local/lib/python3.7/site-packages/esrally/racecontrol.py", line 302, in benchmark_only
return race(cfg, external=True)
File "/usr/local/lib/python3.7/site-packages/esrally/racecontrol.py", line 254, in race
raise exceptions.RallyError(result.message, result.cause)
esrally.exceptions.RallyError: ('Could not launch cluster', Elasticsearch REST API layer is not available. Forcefully terminated cluster.)
2

Hello,

This question looks identical to the previous one you opened: Rally cannot start.

I've now closed Rally cannot start and will update the description here with the additional logs from the other topic.

Dimitris

3

@vishal_viswanathan to your question, what is your Rally command and are you trying to benchmark an existing cluster or have Rally set it up for you? If the former you should take a look a the benchmark-only pipeline.

4

@dliappis this is the error I am getting

rally@d7733066d34e:~$ esrally --track=pmc --target-hosts=IP:9300,IP:9300,IP:9300 --pipeline=benchmark-only --distribution-version=6.8.2

____        ____

/ __ ____ / / / __
/ // / __ `/ / / / / /
/ , / // / / / // /
// ||_,///_, /
/____/

[WARNING] No Internet connection detected. Automatic download of track data sets etc. is disabled.

************** WARNING: A dark dungeon lies ahead of you **************

Rally does not have control over the configuration of the benchmarked
Elasticsearch cluster.

Be aware that results may be misleading due to problems with the setup.
Rally is also not able to gather lots of metrics at all (like CPU usage
of the benchmarked cluster) or may even produce misleading metrics (like
the index size).

****** Use this pipeline only if you are aware of the tradeoffs. ******
*************************** Watch your step! ***************************

[WARNING] Ensure that these settings are defined in elasticsearch.yml:

{
"search.default_search_timeout": "-1"
}

If they are absent, running this track will fail or lead to unexpected results.

[ERROR] Cannot race. Could not launch cluster
Elasticsearch REST API layer is not available. Forcefully terminated cluster.

Getting further help:

[INFO] FAILURE (took 58 seconds)

can you take a look and provide a valuable solution?

5

Hi,

You shouldn't be specifying --distribution-version in conjunction with --pipeline=benchmark-only; you are benchmarking an external cluster which has been provisioned already so it has whatever version it has (and Rally will discover the version and store it in the metric store if configured for future reference).

I raised an issue to make this clearer, at the least through the docs: https://github.com/elastic/rally/issues/833

Rgs,
Dimitris

6

Also note you are using port 9300 which, by default, is the transport port; the port you should be using for Elasticsearch is the http port instead, typically 9200.

7

@dliappis

Getting below rally@d7733066d34e:~$ esrally --track=pmc --target-hosts=IP:9200,IP:9200,IP:9200 --pipeline=benchmark-only

____        ____

/ __ ____ / / / __
/ // / __ `/ / / / / /
/ , / // / / / // /
// ||_,///_, /
/____/

[WARNING] No Internet connection detected. Automatic download of track data sets etc. is disabled.

************** WARNING: A dark dungeon lies ahead of you **************

Rally does not have control over the configuration of the benchmarked
Elasticsearch cluster.

Be aware that results may be misleading due to problems with the setup.
Rally is also not able to gather lots of metrics at all (like CPU usage
of the benchmarked cluster) or may even produce misleading metrics (like
the index size).

****** Use this pipeline only if you are aware of the tradeoffs. ******
*************************** Watch your step! ***************************

[ERROR] Cannot race. Error in race control (('A distribution version is required. Please specify it with --distribution-version.', None))

Getting further help:

[INFO] FAILURE (took 29 seconds)

8

Hi,

  1. What's the Rally version? esrally --version

  2. Can you paste the most recent Rally logs (e.g. tail -100 ~/.rally/logs/rally.log)

  3. Can you confirm you can reach the Elasticsearch HTTP API from the Rally node? e.g. try curl http://IP:9200

9

Hi,

  1. rally@d7733066d34e:~$ esrally --version
    esrally 1.3.0
  2. rally@d7733066d34e:~$ tail -100 ~/.rally/logs/rally.log

File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 333, in increment
raise six.reraise(type(error), error, _stacktrace)

File "/usr/local/lib/python3.7/site-packages/urllib3/packages/six.py", line 686, in reraise
raise value

File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 601, in urlopen
chunked=chunked)

File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 357, in _make_request
conn.request(method, url, **httplib_request_kw)

File "/usr/local/lib/python3.7/http/client.py", line 1229, in request
self._send_request(method, url, body, headers, encode_chunked)

File "/usr/local/lib/python3.7/http/client.py", line 1275, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)

File "/usr/local/lib/python3.7/http/client.py", line 1224, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)

File "/usr/local/lib/python3.7/http/client.py", line 1016, in _send_output
self.send(msg)

File "/usr/local/lib/python3.7/http/client.py", line 956, in send
self.connect()

File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 166, in connect
conn = self._new_conn()

File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 150, in _new_conn
self, "Failed to establish a new connection: %s" % e)

urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fe4c88dcb38>: Failed to establish a new connection: [Errno 113] No route to host

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "/usr/local/lib/python3.7/site-packages/esrally/mechanic/mechanic.py", line 218, in cluster_distribution_version
return es.info()["version"]["number"]

File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/utils.py", line 76, in _wrapped
return func(*args, params=params, **kwargs)

File "/usr/local/lib/python3.7/site-packages/elasticsearch/client/init.py", line 241, in info
return self.transport.perform_request('GET', '/', params=params)

File "/usr/local/lib/python3.7/site-packages/elasticsearch/transport.py", line 314, in perform_request
status, headers_response, data = connection.perform_request(method, url, params, body, headers=headers, ignore=ignore, timeout=timeout)

File "/usr/local/lib/python3.7/site-packages/elasticsearch/connection/http_urllib3.py", line 175, in perform_request
raise ConnectionError('N/A', str(e), e)

elasticsearch.exceptions.ConnectionError: ConnectionError(<urllib3.connection.HTTPConnection object at 0x7fe4c88dcb38>: Failed to establish a new connection: [Errno 113] No route to host) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7fe4c88dcb38>: Failed to establish a new connection: [Errno 113] No route to host)

2019-12-03 12:01:19,965 ActorAddr-(T|:44702)/PID:513 esrally.actor ERROR Error in race control
Traceback (most recent call last):

File "/usr/local/lib/python3.7/site-packages/esrally/actor.py", line 84, in guard
return f(self, msg, sender)

File "/usr/local/lib/python3.7/site-packages/esrally/racecontrol.py", line 119, in receiveMsg_Setup
self.setup(msg, sender)

File "/usr/local/lib/python3.7/site-packages/esrally/racecontrol.py", line 200, in setup
raise exceptions.SystemSetupError("A distribution version is required. Please specify it with --distribution-version.")

esrally.exceptions.SystemSetupError: ('A distribution version is required. Please specify it with --distribution-version.', None)

2019-12-03 12:01:19,970 ActorAddr-(T|:44702)/PID:513 esrally.actor INFO BenchmarkActor received unknown message [ActorExitRequest] (ignoring).
2019-12-03 12:01:22,974 -not-actor-/PID:506 esrally.rally INFO Attempting to shutdown internal actor system.
2019-12-03 12:01:22,977 -not-actor-/PID:512 root INFO ActorSystem Logging Shutdown
2019-12-03 12:01:22,998 -not-actor-/PID:511 root INFO ---- Actor System shutdown
2019-12-03 12:01:22,999 -not-actor-/PID:506 esrally.rally INFO Actor system is still running. Waiting...
2019-12-03 12:01:24,0 -not-actor-/PID:506 esrally.rally INFO Shutdown completed.
2019-12-03 12:01:24,0 -not-actor-/PID:506 esrally.rally ERROR Cannot run subcommand [race].
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/esrally/rally.py", line 515, in dispatch_sub_command
race(cfg)
File "/usr/local/lib/python3.7/site-packages/esrally/rally.py", line 433, in race
with_actor_system(lambda c: racecontrol.run(c), cfg)
File "/usr/local/lib/python3.7/site-packages/esrally/rally.py", line 461, in with_actor_system
runnable(cfg)
File "/usr/local/lib/python3.7/site-packages/esrally/rally.py", line 433, in
with_actor_system(lambda c: racecontrol.run(c), cfg)
File "/usr/local/lib/python3.7/site-packages/esrally/racecontrol.py", line 369, in run
raise e
File "/usr/local/lib/python3.7/site-packages/esrally/racecontrol.py", line 366, in run
pipeline(cfg)
File "/usr/local/lib/python3.7/site-packages/esrally/racecontrol.py", line 78, in call
self.target(cfg)
File "/usr/local/lib/python3.7/site-packages/esrally/racecontrol.py", line 302, in benchmark_only
return race(cfg, external=True)
File "/usr/local/lib/python3.7/site-packages/esrally/racecontrol.py", line 254, in race
raise exceptions.RallyError(result.message, result.cause)
esrally.exceptions.RallyError: ("Error in race control (('A distribution version is required. Please specify it with --distribution-version.', None))", None)
3) I have installed the docker ally container on one of the ES node. There is no seperate node.

10

So logs tell us that Rally is unable to connect to the provided host:port list:

elasticsearch.exceptions.ConnectionError: ConnectionError(<urllib3.connection.HTTPConnection object at 0x7fe4c88dcb38>: Failed to establish a new connection: [Errno 113] No route to host) caused by: NewConnectionError(<urllib3.connection.HTTPConnection object at 0x7fe4c88dcb38>: Failed to establish a new connection: [Errno 113] No route to host)

I suggest you confirm connectivity from the Rally host as indicated in step 3. in the last comment.

11

@dliappis So for running rally as docker container we need to open a port?

12

You need to confirm that wherever Rally runs on (i.e. inside a real OS, VM or a container) you are able to connect to the IP:port of the Elasticsearch nodes.

13

I am running rally inside the docker container. I don't have any ping or nc command to check the connectivity to ES nodes. Any suggestions?

14

Here are results

From rally docker conatiner, I am able to connect to three test node.

rally@d7733066d34e:~ curl -k -u elastic:******* "https://*.*.*.*:9200" { "name" : "c4_testelk_node_02", "cluster_name" : "c4_testelk", "cluster_uuid" : "Lw94O89STI6OoFHJEcbGTw", "version" : { "number" : "6.8.2", "build_flavor" : "default", "build_type" : "docker", "build_hash" : "b506955", "build_date" : "2019-07-24T15:24:41.545295Z", "build_snapshot" : false, "lucene_version" : "7.7.0", "minimum_wire_compatibility_version" : "5.6.0", "minimum_index_compatibility_version" : "5.0.0" }, "tagline" : "You Know, for Search" } rally@d7733066d34e:~ curl -k -u elastic:******* "https://...:9200"
{
"name" : "c4_testelk_node_03",
"cluster_name" : "c4_testelk",
"cluster_uuid" : "Lw94O89STI6OoFHJEcbGTw",
"version" : {
"number" : "6.8.2",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "b506955",
"build_date" : "2019-07-24T15:24:41.545295Z",
"build_snapshot" : false,
"lucene_version" : "7.7.0",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}
rally@d7733066d34e:~$ curl -k -u elastic:******* "https://...:9200"
{
"name" : "c4_testelk_node_01",
"cluster_name" : "c4_testelk",
"cluster_uuid" : "Lw94O89STI6OoFHJEcbGTw",
"version" : {
"number" : "6.8.2",
"build_flavor" : "default",
"build_type" : "docker",
"build_hash" : "b506955",
"build_date" : "2019-07-24T15:24:41.545295Z",
"build_snapshot" : false,
"lucene_version" : "7.7.0",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}

15

So from your last reply I see you are accessing the Elasticsearch cluster with https including authentication.

Your earlier reply to how you run Rally only specified target-hosts.

Please take a look in the client-options documentation page to see how to specify credentials, use TLS and not verify certificates (which is what you are doing when you specify -k in curl).

16

After running with the below authentication, I am getting below error

rally@d7733066d34e:~$ esrally --track=pmc --target-hosts=IP:9200,IP:9200,IP:9200 --pipeline=benchmark-only --client-options="use_ssl:true,verify_certs:false,basic_auth_user:'elastic',basic_auth_password:'******'"

____        ____

/ __ ____ / / / __
/ // / __ `/ / / / / /
/ , / // / / / // /
// ||_,///_, /
/____/

[INFO] You did not provide an explicit timeout in the client options. Assuming default of 10 seconds.
[WARNING] No Internet connection detected. Automatic download of track data sets etc. is disabled.

************** WARNING: A dark dungeon lies ahead of you **************

Rally does not have control over the configuration of the benchmarked
Elasticsearch cluster.

Be aware that results may be misleading due to problems with the setup.
Rally is also not able to gather lots of metrics at all (like CPU usage
of the benchmarked cluster) or may even produce misleading metrics (like
the index size).

****** Use this pipeline only if you are aware of the tradeoffs. ******
*************************** Watch your step! ***************************

[WARNING] Ensure that these settings are defined in elasticsearch.yml:

{
"search.default_search_timeout": "-1"
}

If they are absent, running this track will fail or lead to unexpected results.
[INFO] Racing on track [pmc], challenge [append-no-conflicts] and car ['external'] with version [6.8.2].

[WARNING] indexing_total_time is 1266 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] refresh_total_time is 3787 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[WARNING] flush_total_time is 10141 ms indicating that the cluster is not in a defined clean state. Recorded index time metrics may be misleading.
[ERROR] Cannot race. Error in track preparator (('Cannot find /rally/.rally/benchmarks/data/pmc/documents.json.bz2. Please disable offline mode and retry again.', None))

I cannot run the pipeline bench in offline mode as the docker images from rally doesn't allow as it will only run in pipleine-only mode.

Also I am not able to find the below folder inside docker

/rally/.rally/benchmarks/data/pmc/documents.json.bz2.

17

Hi,

Rally needs to download the necessary tracks (in your case pmc) and it appears it doesn't have access to the internet (from within your container at least) based on the error message:

[WARNING] No Internet connection detected. Automatic download of track data sets etc. is disabled.

Please take a look at the Offline usage section where you will instructions on downloading the tracks yourself (e.g. using this offline script) and then ensure you are bind mounting them in the Docker container as described in the docs.

18

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.