Max retries exceeded with url: /pmc/documents.json.bz2 SSLCertVerificationError [SSL: CERTIFICATE_VERIFY_FAILED]

Swathi_Kakumanu · June 11, 2023, 5:24am

Hi,

I am facing an issue with SSL Verification of esrally for rally-tracks.elastic.co', port=443 , however curl request and openssl works fine.

FYI, internet works on the rally VM.

Command used to run:

esrally race --track=pmc --pipeline=benchmark-only --target-hosts=192.168.105.116:9200 --client-options="timeout:60,use_ssl:true,verify_certs:true,ca_certs:'/home/administrator/ca.crt',basic_auth_user:'elastic',basic_auth_password:'xxxxx'"

Rally ini file

[meta]
config.version = 17

[system]
env.name = local

[node]
root.dir = /home/administrator/.rally/benchmarks
src.root.dir = /home/administrator/.rally/benchmarks/src

[source]
remote.repo.url = GitHub - elastic/elasticsearch: Free and Open, Distributed, RESTful Search Engine
elasticsearch.src.subdir = elasticsearch

[benchmarks]
local.dataset.cache = /home/administrator/.rally/benchmarks/data

[reporting]
datastore.type = in-memory
#datastore.type = elasticsearch
datastore.host = 192.168.105.116
datastore.port = 9200
datastore.secure = true
datastore.ssl.verification_mode = none
datastore.user =
#datastore.ssl.certificate_authorities = /home/administrator/
datastore.password =

[tracks]
default.url = GitHub - elastic/rally-tracks: Track specifications for the Elasticsearch benchmarking tool Rally

[teams]
default.url = GitHub - elastic/rally-teams: Default Elasticsearch configurations for the Elasticsearch benchmarking tool Rally

[defaults]
preserve_benchmark_candidate = false

[distributions]
release.cache = true

Error Message:

[ERROR] Cannot race. Error in task executor
HTTPSConnectionPool(host='rally-tracks.elastic.co', port=443): Max retries exceeded with url: /pmc/documents.json.bz2 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1131)')))

Curl request

administrator@administrator:~$ curl -Ii https://rally-tracks.elastic.co/
HTTP/2 200
x-guploader-uploadid: ADPycdt6naMLkS8bCtIZzbuWyTiRmbb2et5bk6l7rRWF_rQBK6LzJoJncfCLPC7xdKlWbXitmWJ-TuSjhzw5BemV12fWCJnyjuyI
x-goog-metageneration: 6
content-type: application/xml; charset=UTF-8
content-length: 301291
date: Sat, 10 Jun 2023 08:02:48 GMT
expires: Sat, 10 Jun 2023 08:02:48 GMT
cache-control: private, max-age=0
server: UploadServer
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
via: 1.1 google

Rally log file (looks like getting the issue when downloading the data)

2023-06-10 13:44:47,547 -not-actor-/PID:120043 esrally.rally INFO OS [uname_result(system='Linux', node='administrator', release='5.4.0-144-generic', version='#161-Ubuntu SMP Fri Feb 3 14:49:04 UTC 2023', machine='x86_64', processor='x86_64')]
2023-06-10 13:44:47,547 -not-actor-/PID:120043 esrally.rally INFO Python [namespace(_multiarch='x86_64-linux-gnu', cache_tag='cpython-38', hexversion=50859504, name='cpython', version=sys.version_info(major=3, minor=8, micro=13, releaselevel='final', serial=0))]
2023-06-10 13:44:47,547 -not-actor-/PID:120043 esrally.rally INFO Rally version [2.7.1]
2023-06-10 13:44:47,547 -not-actor-/PID:120043 esrally.utils.net INFO Connecting directly to the Internet (no proxy support) for [all_proxy].
2023-06-10 13:44:47,548 -not-actor-/PID:120043 esrally.utils.net INFO Connecting directly to the Internet (no proxy support) for [all_proxy].
2023-06-10 13:44:47,548 -not-actor-/PID:120043 esrally.rally INFO Cleaning track dependency directory [/home/administrator/.rally/libs]...
2023-06-10 13:44:47,628 -not-actor-/PID:120043 esrally.rally INFO Actor system already running locally? [False]
2023-06-10 13:44:47,629 -not-actor-/PID:120043 esrally.actor INFO Starting actor system with system base [multiprocTCPBase] and capabilities [{'coordinator': True, 'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900'}].
2023-06-10 13:44:47,647 -not-actor-/PID:120052 root INFO ++++ Actor System gen (3, 10) started, admin @ ActorAddr-(T|:1900)
2023-06-10 13:44:47,669 -not-actor-/PID:120043 esrally.racecontrol INFO Race id is [35999042-32df-4e99-9857-c579cbf6b5c6]
2023-06-10 13:44:47,669 -not-actor-/PID:120043 esrally.racecontrol INFO User specified pipeline [benchmark-only].
2023-06-10 13:44:47,669 -not-actor-/PID:120043 esrally.racecontrol INFO Using configured hosts [{'host': '192.168.105.116', 'port': 9200}]
2023-06-10 13:44:47,672 ActorAddr-(T|:1900)/PID:120052 esrally.actor DEBUG Capabilities [{'coordinator': True, 'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900', 'Thespian ActorSystem Name': 'multiprocTCPBase', 'Thespian ActorSystem Version': 2, 'Thespian Watch Supported': True, 'Python Version': (3, 8, 13, 'final', 0), 'Thespian Generation': (3, 10), 'Thespian Version': '1686404687636'}] match requirements [{'coordinator': True}].
2023-06-10 13:44:47,692 ActorAddr-(T|:39433)/PID:120054 esrally.client.factory INFO Creating ES client connected to [{'host': '192.168.105.116', 'port': 9200}] with options [{'timeout': 60, 'use_ssl': True, 'verify_certs': True, 'ca_certs': '/home/administrator/ca.crt', 'basic_auth_user': 'elastic', 'basic_auth_password': '*****'}]
2023-06-10 13:44:47,790 ActorAddr-(T|:39433)/PID:120054 esrally.racecontrol INFO Automatically derived distribution version [8.8.0]
2023-06-10 13:44:52,863 ActorAddr-(T|:39433)/PID:120054 esrally.utils.repo INFO Checking out [8.7] in [/home/administrator/.rally/benchmarks/tracks/default] for distribution version [8.8.0].
2023-06-10 13:44:52,885 ActorAddr-(T|:39433)/PID:120054 esrally.utils.process INFO Already on '8.7'
Your branch is up to date with 'origin/8.7'.

2023-06-10 13:44:52,887 ActorAddr-(T|:39433)/PID:120054 esrally.utils.repo INFO Rebasing on [8.7] in [/home/administrator/.rally/benchmarks/tracks/default] for distribution version [8.8.0].
2023-06-10 13:44:52,911 ActorAddr-(T|:39433)/PID:120054 esrally.utils.process INFO Already on '8.7'
Your branch is up to date with 'origin/8.7'.

2023-06-10 13:44:52,922 ActorAddr-(T|:39433)/PID:120054 esrally.utils.process INFO Current branch 8.7 is up to date.

2023-06-10 13:44:52,936 ActorAddr-(T|:39433)/PID:120054 esrally.track.loader INFO Reading track specification file [/home/administrator/.rally/benchmarks/tracks/default/pmc/track.json].
2023-06-10 13:44:52,975 ActorAddr-(T|:39433)/PID:120054 esrally.track.loader INFO Final rendered track for '/home/administrator/.rally/benchmarks/tracks/default/pmc/track.json' has been written to '/tmp/tmpil1ricbn.json'.
2023-06-10 13:44:52,984 ActorAddr-(T|:39433)/PID:120054 esrally.track.loader INFO Loading template [definition for index pmc in index.json].
2023-06-10 13:44:52,988 ActorAddr-(T|:39433)/PID:120054 esrally.utils.modules INFO Loading component [pmc] from [/home/administrator/.rally/benchmarks/tracks/default/pmc]
2023-06-10 13:44:52,990 ActorAddr-(T|:39433)/PID:120054 esrally.metrics INFO Creating in-memory metrics store
2023-06-10 13:44:52,990 ActorAddr-(T|:39433)/PID:120054 esrally.metrics INFO Opening metrics store for race timestamp=[20230610T134447Z], track=[pmc], challenge=[append-no-conflicts], car=[['external']]
2023-06-10 13:44:52,990 ActorAddr-(T|:39433)/PID:120054 esrally.metrics INFO Creating file race store
2023-06-10 13:44:52,990 ActorAddr-(T|:39433)/PID:120054 esrally.actor INFO Asking mechanic to start the engine.
2023-06-10 13:44:53,6 ActorAddr-(T|:35993)/PID:120073 esrally.actor INFO Received signal from race control to start engine.
2023-06-10 13:44:53,8 ActorAddr-(T|:39433)/PID:120054 esrally.actor INFO Mechanic has started engine successfully.
2023-06-10 13:44:53,6 ActorAddr-(T|:35993)/PID:120073 esrally.actor INFO Cluster will not be provisioned by Rally.
2023-06-10 13:44:53,9 ActorAddr-(T|:39433)/PID:120054 esrally.actor INFO Telling driver to prepare for benchmarking.
2023-06-10 13:44:53,24 ActorAddr-(T|:34865)/PID:120074 esrally.metrics INFO Creating in-memory metrics store
2023-06-10 13:44:53,25 ActorAddr-(T|:34865)/PID:120074 esrally.metrics INFO Opening metrics store for race timestamp=[20230610T134447Z], track=[pmc], challenge=[append-no-conflicts], car=[['external']]
2023-06-10 13:44:53,25 ActorAddr-(T|:34865)/PID:120074 esrally.client.factory INFO Creating ES client connected to [{'host': '192.168.105.116', 'port': 9200}] with options [{'timeout': 60, 'use_ssl': True, 'verify_certs': True, 'ca_certs': '/home/administrator/ca.crt', 'basic_auth_user': 'elastic', 'basic_auth_password': '*****', 'retry-on-timeout': True}]
2023-06-10 13:44:53,27 ActorAddr-(T|:34865)/PID:120074 esrally.driver.driver INFO Checking if REST API is available.
2023-06-10 13:44:53,114 ActorAddr-(T|:34865)/PID:120074 esrally.driver.driver INFO REST API is available.
2023-06-10 13:44:53,117 ActorAddr-(T|:34865)/PID:120074 esrally.actor INFO Starting prepare track process on hosts [['localhost']]
2023-06-10 13:44:53,132 ActorAddr-(T|:38863)/PID:120075 esrally.actor INFO Track Preparator started
2023-06-10 13:44:54,54 ActorAddr-(T|:38863)/PID:120075 esrally.track.loader INFO Reading track specification file [/home/administrator/.rally/benchmarks/tracks/default/pmc/track.json].
2023-06-10 13:44:54,89 ActorAddr-(T|:38863)/PID:120075 esrally.track.loader INFO Final rendered track for '/home/administrator/.rally/benchmarks/tracks/default/pmc/track.json' has been written to '/tmp/tmp0f5613wg.json'.
2023-06-10 13:44:54,97 ActorAddr-(T|:38863)/PID:120075 esrally.track.loader INFO Loading template [definition for index pmc in index.json].
2023-06-10 13:44:54,102 ActorAddr-(T|:38863)/PID:120075 esrally.utils.modules INFO Loading component [pmc] from [/home/administrator/.rally/benchmarks/tracks/default/pmc]
2023-06-10 13:44:54,106 ActorAddr-(T|:38863)/PID:120075 esrally.actor INFO Preparing track [pmc]
2023-06-10 13:44:54,106 ActorAddr-(T|:38863)/PID:120075 esrally.actor INFO Reloading track [pmc] to ensure plugins are up-to-date.
2023-06-10 13:44:54,971 ActorAddr-(T|:38863)/PID:120075 esrally.track.loader INFO Reading track specification file [/home/administrator/.rally/benchmarks/tracks/default/pmc/track.json].
2023-06-10 13:44:55,6 ActorAddr-(T|:38863)/PID:120075 esrally.track.loader INFO Final rendered track for '/home/administrator/.rally/benchmarks/tracks/default/pmc/track.json' has been written to '/tmp/tmpw11nztxy.json'.
2023-06-10 13:44:55,14 ActorAddr-(T|:38863)/PID:120075 esrally.track.loader INFO Loading template [definition for index pmc in index.json].
2023-06-10 13:44:55,18 ActorAddr-(T|:38863)/PID:120075 esrally.utils.modules INFO Loading component [pmc] from [/home/administrator/.rally/benchmarks/tracks/default/pmc]
2023-06-10 13:44:55,972 ActorAddr-(T|:35919)/PID:120103 esrally.utils.modules INFO Loading component [pmc] from [/home/administrator/.rally/benchmarks/tracks/default/pmc]
2023-06-10 13:44:55,972 ActorAddr-(T|:45841)/PID:120106 esrally.utils.modules INFO Loading component [pmc] from [/home/administrator/.rally/benchmarks/tracks/default/pmc]
2023-06-10 13:44:55,847 ActorAddr-(T|:38863)/PID:120075 esrally.utils.modules INFO Loading component [pmc] from [/home/administrator/.rally/benchmarks/tracks/default/pmc]
2023-06-10 13:44:55,973 ActorAddr-(T|:44811)/PID:120104 esrally.utils.modules INFO Loading component [pmc] from [/home/administrator/.rally/benchmarks/tracks/default/pmc]
2023-06-10 13:44:55,974 ActorAddr-(T|:37641)/PID:120105 esrally.utils.modules INFO Loading component [pmc] from [/home/administrator/.rally/benchmarks/tracks/default/pmc]
2023-06-10 13:44:55,976 ActorAddr-(T|:42497)/PID:120107 esrally.utils.modules INFO Loading component [pmc] from [/home/administrator/.rally/benchmarks/tracks/default/pmc]
2023-06-10 13:45:06,72 ActorAddr-(T|:41271)/PID:120143 esrally.utils.modules INFO Loading component [pmc] from [/home/administrator/.rally/benchmarks/tracks/default/pmc]
2023-06-10 13:45:06,72 ActorAddr-(T|:33821)/PID:120147 esrally.utils.modules INFO Loading component [pmc] from [/home/administrator/.rally/benchmarks/tracks/default/pmc]
2023-06-10 13:45:06,72 ActorAddr-(T|:38313)/PID:120145 esrally.utils.modules INFO Loading component [pmc] from [/home/administrator/.rally/benchmarks/tracks/default/pmc]
2023-06-10 13:45:06,73 ActorAddr-(T|:39089)/PID:120148 esrally.utils.modules INFO Loading component [pmc] from [/home/administrator/.rally/benchmarks/tracks/default/pmc]
2023-06-10 13:45:06,73 ActorAddr-(T|:33045)/PID:120150 esrally.utils.modules INFO Loading component [pmc] from [/home/administrator/.rally/benchmarks/tracks/default/pmc]
2023-06-10 13:45:06,74 ActorAddr-(T|:39993)/PID:120149 esrally.utils.modules INFO Loading component [pmc] from [/home/administrator/.rally/benchmarks/tracks/default/pmc]
2023-06-10 13:45:06,75 ActorAddr-(T|:35919)/PID:120103 esrally.track.loader INFO Resolved data root directory for document corpus [pmc] in track [pmc] to [['/home/administrator/.rally/benchmarks/data/pmc']].
2023-06-10 13:45:06,76 ActorAddr-(T|:35919)/PID:120103 esrally.track.loader INFO Downloading data from [https://rally-tracks.elastic.co/pmc/documents.json.bz2] (5657 MB) to [/home/administrator/.rally/benchmarks/data/pmc/documents.json.bz2].
> 2023-06-10 13:45:11,80 ActorAddr-(T|:35919)/PID:120103 esrally.driver.driver ERROR Worker failed. Notifying parent...
Traceback (most recent call last):

File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(

File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/connectionpool.py", line 386, in _make_request
self._validate_conn(conn)

File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1040, in _validate_conn
conn.connect()

File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/connection.py", line 414, in connect
self.sock = ssl_wrap_socket(

File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 449, in ssl_wrap_socket
ssl_sock = _ssl_wrap_socket_impl(

File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 493, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)

File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(

File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/ssl.py", line 1040, in _create
self.do_handshake()

File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()

ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1131)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):

File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)

File "/home/administrator/.local/lib/python3.8/site-packages/esrally/track/loader.py", line 457, in prepare_docs
preparator.prepare_document_set(document_set, data_root[0])

File "/home/administrator/.local/lib/python3.8/site-packages/esrally/track/loader.py", line 620, in prepare_document_set
self.downloader.download(document_set.base_url, target_path, expected_size)

File "/home/administrator/.local/lib/python3.8/site-packages/esrally/track/loader.py", line 528, in download
net.download(data_url, target_path, size_in_bytes, progress_indicator=progress)

File "/home/administrator/.local/lib/python3.8/site-packages/esrally/utils/net.py", line 260, in download
expected_size_in_bytes = download_http(url, tmp_data_set_path, expected_size_in_bytes, progress_indicator)

File "/home/administrator/.local/lib/python3.8/site-packages/esrally/utils/net.py", line 217, in download_http
return _download_http(url, local_path, expected_size_in_bytes, progress_indicator)

File "/home/administrator/.local/lib/python3.8/site-packages/esrally/utils/net.py", line 190, in _download_http
with _request(

File "/home/administrator/.local/lib/python3.8/site-packages/esrally/utils/net.py", line 287, in _request
return manager.request(method, url, **kwargs)

File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/request.py", line 74, in request
return self.request_encode_url(

File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/request.py", line 96, in request_encode_url
return self.urlopen(method, url, **extra_kw)

File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/poolmanager.py", line 376, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)

File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/connectionpool.py", line 813, in urlopen
return self.urlopen(

File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/connectionpool.py", line 813, in urlopen
return self.urlopen(

File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/connectionpool.py", line 813, in urlopen
return self.urlopen(

[Previous line repeated 7 more times]

File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/connectionpool.py", line 785, in urlopen
retries = retries.increment(

File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))

urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='rally-tracks.elastic.co', port=443): Max retries exceeded with url: /pmc/documents.json.bz2 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1131)')))

2023-06-10 13:45:11,91 ActorAddr-(T|:34865)/PID:120074 esrally.actor ERROR Main driver received a fatal exception from a load generator. Shutting down.
2023-06-10 13:45:11,91 ActorAddr-(T|:34865)/PID:120074 esrally.metrics INFO Closing metrics store.
2023-06-10 13:45:11,96 -not-actor-/PID:120043 esrally.racecontrol ERROR A benchmark failure has occurred
2023-06-10 13:45:11,97 -not-actor-/PID:120043 esrally.racecontrol INFO Telling benchmark actor to exit.
2023-06-10 13:45:11,94 ActorAddr-(T|:39433)/PID:120054 esrally.actor INFO Received a benchmark failure from [ActorAddr-(T|:34865)] and will forward it now.
2023-06-10 13:45:11,100 ActorAddr-(T|:34865)/PID:120074 esrally.actor INFO Main driver received ActorExitRequest and will terminate all load generators.
2023-06-10 13:45:14,101 -not-actor-/PID:120043 esrally.rally INFO Attempting to shutdown internal actor system.
2023-06-10 13:45:14,104 -not-actor-/PID:120053 root INFO ActorSystem Logging Shutdown
2023-06-10 13:45:14,126 -not-actor-/PID:120052 root INFO ---- Actor System shutdown
2023-06-10 13:45:14,126 -not-actor-/PID:120043 esrally.rally INFO Actor system is still running. Waiting...
2023-06-10 13:45:15,128 -not-actor-/PID:120043 esrally.rally INFO Shutdown completed.
2023-06-10 13:45:15,128 -not-actor-/PID:120043 esrally.rally ERROR Cannot run subcommand [race].
Traceback (most recent call last):
File "/home/administrator/.local/lib/python3.8/site-packages/esrally/rally.py", line 1171, in dispatch_sub_command
race(cfg, args.kill_running_processes)
File "/home/administrator/.local/lib/python3.8/site-packages/esrally/rally.py", line 919, in race
with_actor_system(racecontrol.run, cfg)
File "/home/administrator/.local/lib/python3.8/site-packages/esrally/rally.py", line 949, in with_actor_system
runnable(cfg)
File "/home/administrator/.local/lib/python3.8/site-packages/esrally/racecontrol.py", line 372, in run
raise e
File "/home/administrator/.local/lib/python3.8/site-packages/esrally/racecontrol.py", line 369, in run
pipeline(cfg)
File "/home/administrator/.local/lib/python3.8/site-packages/esrally/racecontrol.py", line 71, in call
self.target(cfg)
File "/home/administrator/.local/lib/python3.8/site-packages/esrally/racecontrol.py", line 308, in benchmark_only
return race(cfg, external=True)
File "/home/administrator/.local/lib/python3.8/site-packages/esrally/racecontrol.py", line 266, in race
raise exceptions.RallyError(result.message, result.cause)
esrally.exceptions.RallyError: Error in task executor

Can someone guide me here?

Quentin_Pradet · June 12, 2023, 6:48am

Hello, and thanks for your interest in Rally.

The issue here is that you're telling Rally to use /home/administrator/ca.crt for certificate authorities and nothing else. But https://rally-tracks.elastic.co/ needs a different certificate authority, one of the defaults one from the certifi · PyPI package.

I'm seeing two options here:

Since you're on Ubuntu, follow Installing a root CA certificate in the trust store | Ubuntu to add you local ca.crt file to the global store, and then configure ca_certs to /etc/ssl/certs/ca-certificates.crt
Download tracks separately using the offline mode: Offline Usage - Rally 2.7.1 documentation

Quentin_Pradet · June 12, 2023, 12:49pm

Actually, downloading tracks does not use the --client-options configuration, so this is puzzling. Can you please run the following script? This is what Rally does under the hood. This will help diagnose the issue.

import certifi
import urllib3
import esrally


print(esrally.__version__, urllib3.__version__, certifi.__version__)
http = urllib3.PoolManager(cert_reqs="CERT_REQUIRED", ca_certs=certifi.where())
http.request("GET", "https://rally-tracks.elastic.co")
response = http.request("GET", "https://rally-tracks.elastic.co")
assert response.status == 200

Swathi_Kakumanu · June 13, 2023, 1:45pm

Sure, this the output:

administrator@administrator:~$ python3 rally_http_test.py

2.7.1 1.26.9 2023.05.07
Traceback (most recent call last):
File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen
httplib_response = self._make_request(
File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/connectionpool.py", line 386, in _make_request
self._validate_conn(conn)
File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1040, in validate_conn
conn.connect()
File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/connection.py", line 414, in connect
self.sock = ssl_wrap_socket(
File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/util/ssl.py", line 449, in ssl_wrap_socket
ssl_sock = ssl_wrap_socket_impl(
File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/util/ssl.py", line 493, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/ssl.py", line 1040, in _create
self.do_handshake()
File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1131)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "rally_http_test.py", line 8, in
http.request("GET", "https://rally-tracks.elastic.co")
File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/request.py", line 74, in request
return self.request_encode_url(
File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/request.py", line 96, in request_encode_url
return self.urlopen(method, url, **extra_kw)
File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/poolmanager.py", line 376, in urlopen
response = conn.urlopen(method, u.request_uri, **kw)
File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/connectionpool.py", line 813, in urlopen
return self.urlopen(
File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/connectionpool.py", line 813, in urlopen
return self.urlopen(
File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/connectionpool.py", line 813, in urlopen
return self.urlopen(
File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/connectionpool.py", line 785, in urlopen
retries = retries.increment(
File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/util/retry.py", line 592, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='rally-tracks.elastic.co', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1131)')))

Swathi_Kakumanu · June 13, 2023, 2:26pm

What I have observed is:

certifi.where() is showing as /home/administrator/.local/lib/python3.8/site-packages/certifi/cacert.pem and my ca root certificate is in /home/administrator/

from your previous instructions, I performed the below steps:

Copied these two files (ca.crt and cacert.pem) to /usr/local/share/ca-certificates and ran the sudo update-ca-certificates
Changed the ca_certs value in the test python script from ca_certs=certifi.where() to ca_certs='/etc/ssl/certs/ca-certificates.crt'
Executed the python file, this time I didn't get any error. > administrator@administrator:~$ python3 rally_http_test.py

2.7.1 1.26.9 2023.05.07

However, If if I configure this certificate in the esrally command line, then I am still getting the certificate error:

esrally race --pipeline benchmark-only --track=geopoint --challenge append-no-conflicts-index-only --target-hosts=192.168.105.116:9200 --track-params="bulk_indexing_clients:10" --client-options="use_ssl:true,verify_certs:true,ca_certs:'/etc/ssl/certs/ca-certificates.crt',basic_auth_user:'elastic',basic_auth_password:'xxxxx'"

[ERROR] Cannot race. Error in task executor
HTTPSConnectionPool(host='rally-tracks.elastic.co', port=443): Max retries exceeded with url: /geopoint/documents.json.bz2 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1131)')))

Is there any other way to provide this value to the esrally command? is this the right direction?

Quentin_Pradet · June 15, 2023, 8:53am

Yes, this is the correct direction. It looks like your Rally VM is behind a MITM SSL proxy which means you need a different SSL certificate to connect to the Internet. Maybe this is this ca.crt file, maybe this is configured already in the Ubuntu system certificates, it’s hard to tell from your experiment.

Unfortunately Rally does not allow configuring the certificate authorities used to download tracks: rally/esrally/utils/net.py at 675f1bdbe1fb96f07fefd947a1cd316525544b3a · elastic/rally · GitHub

But again you can use offline mode by downloading the tracks separately and the sending them to the Rally VM: Offline Usage - Rally 2.8.0 documentation

(Thanks @sethmlarson for the help here)

Swathi_Kakumanu · June 19, 2023, 5:04am

Thanks for the guidance Quentin_Pradet

Swathi_Kakumanu · July 5, 2023, 3:51pm

Hi @Quentin_Pradet , as a workaround for online execution, I did the below changes.

1st Step : Updating the cacerts

sudo cp ca.crt /usr/local/share/ca-certificates
sudo cp /home/administrator/.local/lib/python3.8/site-packages/certifi/cacert.pem /usr/local/share/ca-certificates
sudo update-ca-certificates

which will update the '/etc/ssl/certs/ca-certificates.crt'

2nd Step: Replacing the default certifi file with the updated one:

Verify certifi.where() path, in my case it is /home/administrator/.local/lib/python3.8/site-packages/certifi/cacert.pem
Take a backup of this file just for reference
Copy the file /etc/ssl/certs/ca-certificates.crt as cacert.pem and replace in the path /home/administrator/.local/lib/python3.8/site-packages/certifi/

So that certifi will find this certificate which has external access now. Basically replacing the default file.

With these steps I was able to execute online mode.

Just sharing it here so that it can help others.

system · August 2, 2023, 3:52pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Esrally SSL error certificate verify failed Elasticsearch rally	8	2138	December 10, 2020
Client-options "verify_certs:false" seems to be ignored Elasticsearch rally	1	566	July 15, 2020
Esrally-elastic mutual TLS Elasticsearch rally	19	1775	September 18, 2018
ConnectionRefusedError: [Errno 111] Connection refused error while running rally for an existing elastic cluster Elasticsearch rally	4	856	July 10, 2023
Rally test failed with SSL error Elasticsearch rally	7	1082	June 18, 2019

Max retries exceeded with url: /pmc/documents.json.bz2 SSLCertVerificationError [SSL: CERTIFICATE_VERIFY_FAILED]

Related Topics