Hello Elasticworld,
I'm striving to run a race on my test cluster, but it's apparently not able to connect to ES
I'm running 2 ES 6.8.18 nodes running on RH 7.9
rally runs from RH 8.4
[esrally@uttest01 ~]$ esrally --version
esrally 2.3.0
[esrally@uttest01 ~]$ curl -u elastic:<redacted> -X GET 'http://utlogdb01.vlhll.loc:9200/_cluster/health?pretty' {
"cluster_name" : "graylog-test",
"status" : "green",
"timed_out" : false,
"number_of_nodes" : 2,
"number_of_data_nodes" : 2,
[...]
}
[esrally@uttest01 ~]$ esrally race --distribution-version=6.8.18 --track=nested --challenge=nested-search-challenge --report-file=/tmp/esrally-report.md --target-hosts=utlogdb01.vlhll.loc:9200 --pipeline=benchmark-only --client-options="use_ssl:true,verify_certs:False,basic_auth_user:'elastic',basic_auth_password:'<redacted>'"
____ ____
/ __ \____ _/ / /_ __
[...]
[WARNING] No Internet connection detected. Automatic download of track data sets etc. is disabled.
[INFO] Race id is [41c963d6-a226-41f8-a5e4-1e1464067c2c]
[ERROR] Cannot race. Traceback (most recent call last):
File "/home/esrally/.local/lib/python3.8/site-packages/esrally/actor.py", line 92, in guard
return f(self, msg, sender)
File "/home/esrally/.local/lib/python3.8/site-packages/esrally/driver/driver.py", line 269, in receiveMsg_PrepareBenchmark
self.coordinator.prepare_benchmark(msg.track)
File "/home/esrally/.local/lib/python3.8/site-packages/esrally/driver/driver.py", line 655, in prepare_benchmark
self.wait_for_rest_api(es_clients)
File "/home/esrally/.local/lib/python3.8/site-packages/esrally/driver/driver.py", line 626, in wait_for_rest_api
raise exceptions.SystemSetupError("Elasticsearch REST API layer is not available.")
esrally.exceptions.SystemSetupError: Elasticsearch REST API layer is not available.
Getting further help:
*********************
* Check the log files in /home/esrally/.rally/logs for errors.
* Read the documentation at https://esrally.readthedocs.io/en/2.3.0/.
* Ask a question on the forum at https://discuss.elastic.co/tags/c/elastic-stack/elasticsearch/rally.
* Raise an issue at https://github.com/elastic/rally/issues and include the log files in /home/esrally/.rally/logs.
---------------------------------
[INFO] FAILURE (took 132 seconds)
---------------------------------
At the end of the log it claims that Elasticsearch REST API layer is not available
2021-11-04 11:52:38,28 -not-actor-/PID:2014740 esrally.rally INFO OS [uname_result(system='Linux', node='utatlassian01', release='4.18.0-305.19.1.el8_4.x86_64', version='#1 SMP Tue Sep 7 07:07:31 EDT 2021', machine='x86_64', processor='x86_64')]
2021-11-04 11:52:38,28 -not-actor-/PID:2014740 esrally.rally INFO Python [namespace(_multiarch='x86_64-linux-gnu', cache_tag='cpython-38', hexversion=50857712, name='cpython', version=sys.version_info(major=3, minor=8, micro=6, releaselevel='final', serial=0))]
2021-11-04 11:52:38,28 -not-actor-/PID:2014740 esrally.rally INFO Rally version [2.3.0]
2021-11-04 11:52:38,28 -not-actor-/PID:2014740 esrally.utils.net INFO Connecting directly to the Internet (no proxy support).
2021-11-04 11:52:46,48 -not-actor-/PID:2014740 esrally.rally WARNING No Internet connection detected. Automatic download of track data sets etc. is disabled.
2021-11-04 11:52:46,66 -not-actor-/PID:2014740 esrally.rally INFO Actor system already running locally? [False]
2021-11-04 11:52:46,66 -not-actor-/PID:2014740 esrally.actor INFO Starting actor system with system base [multiprocTCPBase] and capabilities [{'coordinator': True, 'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900'}].
2021-11-04 11:52:46,79 -not-actor-/PID:2014755 root INFO ++++ Actor System gen (3, 10) started, admin @ ActorAddr-(T|:1900)
2021-11-04 11:52:46,89 -not-actor-/PID:2014740 esrally.racecontrol INFO Race id is [41c963d6-a226-41f8-a5e4-1e1464067c2c]
2021-11-04 11:52:46,89 -not-actor-/PID:2014740 esrally.racecontrol INFO User specified pipeline [benchmark-only].
2021-11-04 11:52:46,89 -not-actor-/PID:2014740 esrally.racecontrol INFO Using configured hosts [{'host': 'utlogdb01.vlhll.loc', 'port': 9200}]
2021-11-04 11:52:46,90 -not-actor-/PID:2014740 esrally.actor INFO Joining already running actor system with system base [multiprocTCPBase].
2021-11-04 11:52:46,91 ActorAddr-(T|:1900)/PID:2014755 esrally.actor INFO Capabilities [{'coordinator': True, 'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900', 'Thespian ActorSystem Name': 'multiprocTCPBase', 'Thespian ActorSystem Version': 2, 'Thespian Watch Supported': True, 'Python Version': (3, 8, 6, 'final', 0), 'Thespian Generation': (3, 10), 'Thespian Version': '1636026766073'}] match requirements [{'coordinator': True}].
2021-11-04 11:52:46,109 ActorAddr-(T|:38951)/PID:2014757 esrally.utils.repo INFO Checking out [6] in [/home/esrally/.rally/benchmarks/tracks/default] for distribution version [6.8.18].
2021-11-04 11:52:46,118 ActorAddr-(T|:38951)/PID:2014757 esrally.utils.process INFO Already on '6'
Your branch is up to date with 'origin/6'.
2021-11-04 11:52:46,118 ActorAddr-(T|:38951)/PID:2014757 esrally.utils.repo INFO Rebasing on [6] in [/home/esrally/.rally/benchmarks/tracks/default] for distribution version [6.8.18].
2021-11-04 11:52:46,129 ActorAddr-(T|:38951)/PID:2014757 esrally.utils.process INFO Already on '6'
Your branch is up to date with 'origin/6'.
2021-11-04 11:52:46,135 ActorAddr-(T|:38951)/PID:2014757 esrally.utils.process INFO Current branch 6 is up to date.
2021-11-04 11:52:46,143 ActorAddr-(T|:38951)/PID:2014757 esrally.track.loader INFO Reading track specification file [/home/esrally/.rally/benchmarks/tracks/default/nested/track.json].
2021-11-04 11:52:46,160 ActorAddr-(T|:38951)/PID:2014757 esrally.track.loader INFO Final rendered track for '/home/esrally/.rally/benchmarks/tracks/default/nested/track.json' has been written to '/tmp/tmpu2dbzg5t.json'.
2021-11-04 11:52:46,168 ActorAddr-(T|:38951)/PID:2014757 esrally.track.loader INFO Loading template [definition for index sonested in index.json].
2021-11-04 11:52:46,182 ActorAddr-(T|:43875)/PID:2014769 esrally.actor INFO Received signal from race control to start engine.
2021-11-04 11:52:46,171 ActorAddr-(T|:38951)/PID:2014757 esrally.utils.modules INFO Loading component [nested] from [/home/esrally/.rally/benchmarks/tracks/default/nested]
2021-11-04 11:52:46,182 ActorAddr-(T|:43875)/PID:2014769 esrally.actor INFO Cluster will not be provisioned by Rally.
2021-11-04 11:52:46,173 ActorAddr-(T|:38951)/PID:2014757 esrally.metrics INFO Creating in-memory metrics store
2021-11-04 11:52:46,173 ActorAddr-(T|:38951)/PID:2014757 esrally.metrics INFO Opening metrics store for race timestamp=[20211104T115238Z], track=[nested], challenge=[nested-search-challenge], car=[['external']]
2021-11-04 11:52:46,173 ActorAddr-(T|:38951)/PID:2014757 esrally.metrics INFO Creating file race store
2021-11-04 11:52:46,173 ActorAddr-(T|:38951)/PID:2014757 esrally.actor INFO Asking mechanic to start the engine.
2021-11-04 11:52:46,173 ActorAddr-(T|:38951)/PID:2014757 esrally.actor INFO Capabilities [{'coordinator': True, 'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900', 'Thespian ActorSystem Name': 'multiprocTCPBase', 'Thespian ActorSystem Version': 2, 'Thespian Watch Supported': True, 'Python Version': (3, 8, 6, 'final', 0), 'Thespian Generation': (3, 10), 'Thespian Version': '1636026766073'}] match requirements [{'coordinator': True}].
2021-11-04 11:52:46,183 ActorAddr-(T|:38951)/PID:2014757 esrally.actor INFO Mechanic has started engine successfully.
2021-11-04 11:52:46,183 ActorAddr-(T|:38951)/PID:2014757 esrally.actor INFO Capabilities [{'coordinator': True, 'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900', 'Thespian ActorSystem Name': 'multiprocTCPBase', 'Thespian ActorSystem Version': 2, 'Thespian Watch Supported': True, 'Python Version': (3, 8, 6, 'final', 0), 'Thespian Generation': (3, 10), 'Thespian Version': '1636026766073'}] match requirements [{'coordinator': True}].
2021-11-04 11:52:46,186 ActorAddr-(T|:38951)/PID:2014757 esrally.actor INFO Telling driver to prepare for benchmarking.
2021-11-04 11:52:46,212 -not-actor-/PID:2014770 elasticsearch WARNING GET https://utlogdb01.vlhll.loc:9200/ [status:N/A request:0.007s]
Traceback (most recent call last):
File "/home/esrally/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "/home/esrally/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 382, in _make_request
self._validate_conn(conn)
[...]
File "/usr/lib64/python3.8/ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
ssl.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1124)
[...]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/esrally/.local/lib/python3.8/site-packages/elasticsearch/connection/http_urllib3.py", line 251, in perform_request
response = self.pool.urlopen(
File "/home/esrally/.local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 755, in urlopen
retries = retries.increment(
[...]
File "/usr/lib64/python3.8/ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "/usr/lib64/python3.8/ssl.py", line 1040, in _create
self.do_handshake()
File "/usr/lib64/python3.8/ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
urllib3.exceptions.SSLError: [SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1124)
2021-11-04 11:52:46,193 ActorAddr-(T|:39363)/PID:2014770 esrally.metrics INFO Creating in-memory metrics store
2021-11-04 11:52:46,194 ActorAddr-(T|:39363)/PID:2014770 esrally.metrics INFO Opening metrics store for race timestamp=[20211104T115238Z], track=[nested], challenge=[nested-search-challenge], car=[['external']]
2021-11-04 11:54:46,555 ActorAddr-(T|:38951)/PID:2014757 esrally.actor INFO Received a benchmark failure from [ActorAddr-(T|:39363)] and will forward it now.
2021-11-04 11:52:46,194 ActorAddr-(T|:39363)/PID:2014770 esrally.client INFO Creating ES client connected to [{'host': 'utlogdb01.vlhll.loc', 'port': 9200}] with options [{'timeout': 60, 'use_ssl': True, 'verify_certs': False, 'basic_auth_user': 'elastic', 'basic_auth_password': '*****', 'retry-on-timeout': True}]
2021-11-04 11:52:46,194 ActorAddr-(T|:39363)/PID:2014770 esrally.client INFO SSL support: on
[...]
2021-11-04 11:52:46,205 ActorAddr-(T|:39363)/PID:2014770 esrally.driver.driver INFO Checking if REST API is available.
2021-11-04 11:54:46,553 ActorAddr-(T|:39363)/PID:2014770 esrally.driver.driver ERROR REST API layer is not yet available. Stopping benchmark.
2021-11-04 11:54:46,554 ActorAddr-(T|:39363)/PID:2014770 esrally.actor ERROR Error in driver
Traceback (most recent call last):
File "/home/esrally/.local/lib/python3.8/site-packages/esrally/actor.py", line 92, in guard
return f(self, msg, sender)
[...]
File "/home/esrally/.local/lib/python3.8/site-packages/esrally/driver/driver.py", line 626, in wait_for_rest_api
raise exceptions.SystemSetupError("Elasticsearch REST API layer is not available.")
esrally.exceptions.SystemSetupError: Elasticsearch REST API layer is not available.
2021-11-04 11:54:46,559 ActorAddr-(T|:39363)/PID:2014770 esrally.actor INFO Main driver received ActorExitRequest and will terminate all load generators.
2021-11-04 11:54:49,561 -not-actor-/PID:2014740 esrally.rally INFO Attempting to shutdown internal actor system.
2021-11-04 11:54:49,563 -not-actor-/PID:2014756 root INFO ActorSystem Logging Shutdown
2021-11-04 11:54:49,583 -not-actor-/PID:2014755 root INFO ---- Actor System shutdown
2021-11-04 11:54:49,584 -not-actor-/PID:2014740 esrally.rally INFO Actor system is still running. Waiting...
2021-11-04 11:54:50,585 -not-actor-/PID:2014740 esrally.rally INFO Shutdown completed.
2021-11-04 11:54:50,585 -not-actor-/PID:2014740 esrally.rally ERROR Cannot run subcommand [race].
Traceback (most recent call last):
File "/home/esrally/.local/lib/python3.8/site-packages/esrally/rally.py", line 975, in dispatch_sub_command
race(cfg, args.kill_running_processes)
[...]
File "/home/esrally/.local/lib/python3.8/site-packages/esrally/driver/driver.py", line 655, in prepare_benchmark
self.wait_for_rest_api(es_clients)
File "/home/esrally/.local/lib/python3.8/site-packages/esrally/driver/driver.py", line 626, in wait_for_rest_api
raise exceptions.SystemSetupError("Elasticsearch REST API layer is not available.")
esrally.exceptions.SystemSetupError: Elasticsearch REST API layer is not available.
Do you have any hint for such a behaviour?
Kind regards,
Gab
