# Max retries exceeded with url: /pmc/documents.json.bz2 SSLCertVerificationError \[SSL: CERTIFICATE\_VERIFY\_FAILED\]

**URL:** <https://discuss.elastic.co/t/max-retries-exceeded-with-url-pmc-documents-json-bz2-sslcertverificationerror-ssl-certificate-verify-failed/335708>\
**Category:** Elasticsearch\
**Tags:** rally\
**Created:** [June 11, 2023, 5:24am UTC](https://discuss.elastic.co/t/max-retries-exceeded-with-url-pmc-documents-json-bz2-sslcertverificationerror-ssl-certificate-verify-failed/335708 "2023-06-11T05:24:44Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![Swathi\_Kakumanu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/swathi_kakumanu/32/117764_2.png) [@Swathi\_Kakumanu](https://discuss.elastic.co/u/Swathi_Kakumanu)\
**Post date:** [June 11, 2023, 5:24am UTC](https://discuss.elastic.co/t/max-retries-exceeded-with-url-pmc-documents-json-bz2-sslcertverificationerror-ssl-certificate-verify-failed/335708/1 "2023-06-11T05:24:44Z")

</div>

Hi,

I am facing an issue with SSL Verification of esrally for [rally-tracks.elastic.co](http://rally-tracks.elastic.co)', port=443 , however curl request and openssl works fine.

FYI, internet works on the rally VM.

**Command used to run:**

`esrally race --track=pmc --pipeline=benchmark-only --target-hosts=192.168.105.116:9200 --client-options="timeout:60,use_ssl:true,verify_certs:true,ca_certs:'/home/administrator/ca.crt',basic_auth_user:'elastic',basic_auth_password:'xxxxx'"`

Rally ini file

> [meta]  
> config.version = 17
> 
> [system]  
> env.name = local
> 
> [node]  
> root.dir = /home/administrator/.rally/benchmarks  
> src.root.dir = /home/administrator/.rally/benchmarks/src
> 
> [source]  
> remote.repo.url = [GitHub - elastic/elasticsearch: Free and Open, Distributed, RESTful Search Engine](https://github.com/elastic/elasticsearch.git)  
> elasticsearch.src.subdir = elasticsearch
> 
> [benchmarks]  
> local.dataset.cache = /home/administrator/.rally/benchmarks/data
> 
> [reporting]  
> datastore.type = in-memory  
> #datastore.type = elasticsearch  
> datastore.host = 192.168.105.116  
> datastore.port = 9200  
> datastore.secure = true  
> datastore.ssl.verification\_mode = none  
> datastore.user =  
> #datastore.ssl.certificate\_authorities = /home/administrator/  
> datastore.password =
> 
> [tracks]  
> default.url = [GitHub - elastic/rally-tracks: Track specifications for the Elasticsearch benchmarking tool Rally](https://github.com/elastic/rally-tracks)
> 
> [teams]  
> default.url = [GitHub - elastic/rally-teams: Default Elasticsearch configurations for the Elasticsearch benchmarking tool Rally](https://github.com/elastic/rally-teams)
> 
> [defaults]  
> preserve\_benchmark\_candidate = false
> 
> [distributions]  
> release.cache = true

Error Message:

> [ERROR] Cannot race. Error in task executor  
> HTTPSConnectionPool(host='[rally-tracks.elastic.co](http://rally-tracks.elastic.co)', port=443): Max retries exceeded with url: /pmc/documents.json.bz2 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE\_VERIFY\_FAILED] certificate verify failed: self signed certificate in certificate chain (\_ssl.c:1131)')))

**Curl request**

> administrator@administrator:~$ curl -Ii [https://rally-tracks.elastic.co/](https://rally-tracks.elastic.co/)  
> HTTP/2 200  
> x-guploader-uploadid: ADPycdt6naMLkS8bCtIZzbuWyTiRmbb2et5bk6l7rRWF\_rQBK6LzJoJncfCLPC7xdKlWbXitmWJ-TuSjhzw5BemV12fWCJnyjuyI  
> x-goog-metageneration: 6  
> content-type: application/xml; charset=UTF-8  
> content-length: 301291  
> date: Sat, 10 Jun 2023 08:02:48 GMT  
> expires: Sat, 10 Jun 2023 08:02:48 GMT  
> cache-control: private, max-age=0  
> server: UploadServer  
> alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000  
> alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000  
> via: 1.1 google

Rally log file (looks like getting the issue when downloading the data)

> 2023-06-10 13:44:47,547 -not-actor-/PID:120043 esrally.rally INFO OS [uname\_result(system='Linux', node='administrator', release='5.4.0-144-generic', version='#161-Ubuntu SMP Fri Feb 3 14:49:04 UTC 2023', machine='x86\_64', processor='x86\_64')]  
> 2023-06-10 13:44:47,547 -not-actor-/PID:120043 esrally.rally INFO Python [namespace(\_multiarch='x86\_64-linux-gnu', cache\_tag='cpython-38', hexversion=50859504, name='cpython', version=sys.version\_info(major=3, minor=8, micro=13, releaselevel='final', serial=0))]  
> 2023-06-10 13:44:47,547 -not-actor-/PID:120043 esrally.rally INFO Rally version [2.7.1]  
> 2023-06-10 13:44:47,547 -not-actor-/PID:120043 [esrally.utils.net](http://esrally.utils.net) INFO Connecting directly to the Internet (no proxy support) for [all\_proxy].  
> 2023-06-10 13:44:47,548 -not-actor-/PID:120043 [esrally.utils.net](http://esrally.utils.net) INFO Connecting directly to the Internet (no proxy support) for [all\_proxy].  
> 2023-06-10 13:44:47,548 -not-actor-/PID:120043 esrally.rally INFO Cleaning track dependency directory [/home/administrator/.rally/libs]...  
> 2023-06-10 13:44:47,628 -not-actor-/PID:120043 esrally.rally INFO Actor system already running locally? [False]  
> 2023-06-10 13:44:47,629 -not-actor-/PID:120043 esrally.actor INFO Starting actor system with system base [multiprocTCPBase] and capabilities [{'coordinator': True, 'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900'}].  
> 2023-06-10 13:44:47,647 -not-actor-/PID:120052 root INFO ++++ Actor System gen (3, 10) started, admin @ ActorAddr-(T|:1900)  
> 2023-06-10 13:44:47,669 -not-actor-/PID:120043 esrally.racecontrol INFO Race id is [35999042-32df-4e99-9857-c579cbf6b5c6]  
> 2023-06-10 13:44:47,669 -not-actor-/PID:120043 esrally.racecontrol INFO User specified pipeline [benchmark-only].  
> 2023-06-10 13:44:47,669 -not-actor-/PID:120043 esrally.racecontrol INFO Using configured hosts [{'host': '192.168.105.116', 'port': 9200}]  
> 2023-06-10 13:44:47,672 ActorAddr-(T|:1900)/PID:120052 esrally.actor DEBUG Capabilities [{'coordinator': True, 'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900', 'Thespian ActorSystem Name': 'multiprocTCPBase', 'Thespian ActorSystem Version': 2, 'Thespian Watch Supported': True, 'Python Version': (3, 8, 13, 'final', 0), 'Thespian Generation': (3, 10), 'Thespian Version': '1686404687636'}] match requirements [{'coordinator': True}].  
> 2023-06-10 13:44:47,692 ActorAddr-(T|:39433)/PID:120054 esrally.client.factory INFO Creating ES client connected to [{'host': '192.168.105.116', 'port': 9200}] with options [{'timeout': 60, 'use\_ssl': True, 'verify\_certs': True, 'ca\_certs': '/home/administrator/ca.crt', 'basic\_auth\_user': 'elastic', 'basic\_auth\_password': '\*\*\*\*\*'}]  
> 2023-06-10 13:44:47,790 ActorAddr-(T|:39433)/PID:120054 esrally.racecontrol INFO Automatically derived distribution version [8.8.0]  
> 2023-06-10 13:44:52,863 ActorAddr-(T|:39433)/PID:120054 esrally.utils.repo INFO Checking out [8.7] in [/home/administrator/.rally/benchmarks/tracks/default] for distribution version [8.8.0].  
> 2023-06-10 13:44:52,885 ActorAddr-(T|:39433)/PID:120054 esrally.utils.process INFO Already on '8.7'  
> Your branch is up to date with 'origin/8.7'.

> 2023-06-10 13:44:52,887 ActorAddr-(T|:39433)/PID:120054 esrally.utils.repo INFO Rebasing on [8.7] in [/home/administrator/.rally/benchmarks/tracks/default] for distribution version [8.8.0].  
> 2023-06-10 13:44:52,911 ActorAddr-(T|:39433)/PID:120054 esrally.utils.process INFO Already on '8.7'  
> Your branch is up to date with 'origin/8.7'.
> 
> 2023-06-10 13:44:52,922 ActorAddr-(T|:39433)/PID:120054 esrally.utils.process INFO Current branch 8.7 is up to date.
> 
> 2023-06-10 13:44:52,936 ActorAddr-(T|:39433)/PID:120054 esrally.track.loader INFO Reading track specification file [/home/administrator/.rally/benchmarks/tracks/default/pmc/track.json].  
> 2023-06-10 13:44:52,975 ActorAddr-(T|:39433)/PID:120054 esrally.track.loader INFO Final rendered track for '/home/administrator/.rally/benchmarks/tracks/default/pmc/track.json' has been written to '/tmp/tmpil1ricbn.json'.  
> 2023-06-10 13:44:52,984 ActorAddr-(T|:39433)/PID:120054 esrally.track.loader INFO Loading template [definition for index pmc in index.json].  
> 2023-06-10 13:44:52,988 ActorAddr-(T|:39433)/PID:120054 esrally.utils.modules INFO Loading component [pmc] from [/home/administrator/.rally/benchmarks/tracks/default/pmc]  
> 2023-06-10 13:44:52,990 ActorAddr-(T|:39433)/PID:120054 esrally.metrics INFO Creating in-memory metrics store  
> 2023-06-10 13:44:52,990 ActorAddr-(T|:39433)/PID:120054 esrally.metrics INFO Opening metrics store for race timestamp=[20230610T134447Z], track=[pmc], challenge=[append-no-conflicts], car=[['external']]  
> 2023-06-10 13:44:52,990 ActorAddr-(T|:39433)/PID:120054 esrally.metrics INFO Creating file race store  
> 2023-06-10 13:44:52,990 ActorAddr-(T|:39433)/PID:120054 esrally.actor INFO Asking mechanic to start the engine.  
> 2023-06-10 13:44:53,6 ActorAddr-(T|:35993)/PID:120073 esrally.actor INFO Received signal from race control to start engine.  
> 2023-06-10 13:44:53,8 ActorAddr-(T|:39433)/PID:120054 esrally.actor INFO Mechanic has started engine successfully.  
> 2023-06-10 13:44:53,6 ActorAddr-(T|:35993)/PID:120073 esrally.actor INFO Cluster will not be provisioned by Rally.  
> 2023-06-10 13:44:53,9 ActorAddr-(T|:39433)/PID:120054 esrally.actor INFO Telling driver to prepare for benchmarking.  
> 2023-06-10 13:44:53,24 ActorAddr-(T|:34865)/PID:120074 esrally.metrics INFO Creating in-memory metrics store  
> 2023-06-10 13:44:53,25 ActorAddr-(T|:34865)/PID:120074 esrally.metrics INFO Opening metrics store for race timestamp=[20230610T134447Z], track=[pmc], challenge=[append-no-conflicts], car=[['external']]  
> 2023-06-10 13:44:53,25 ActorAddr-(T|:34865)/PID:120074 esrally.client.factory INFO Creating ES client connected to [{'host': '192.168.105.116', 'port': 9200}] with options [{'timeout': 60, 'use\_ssl': True, 'verify\_certs': True, 'ca\_certs': '/home/administrator/ca.crt', 'basic\_auth\_user': 'elastic', 'basic\_auth\_password': '\*\*\*\*\*', 'retry-on-timeout': True}]  
> 2023-06-10 13:44:53,27 ActorAddr-(T|:34865)/PID:120074 esrally.driver.driver INFO Checking if REST API is available.  
> 2023-06-10 13:44:53,114 ActorAddr-(T|:34865)/PID:120074 esrally.driver.driver INFO REST API is available.  
> 2023-06-10 13:44:53,117 ActorAddr-(T|:34865)/PID:120074 esrally.actor INFO Starting prepare track process on hosts [['localhost']]  
> 2023-06-10 13:44:53,132 ActorAddr-(T|:38863)/PID:120075 esrally.actor INFO Track Preparator started  
> 2023-06-10 13:44:54,54 ActorAddr-(T|:38863)/PID:120075 esrally.track.loader INFO Reading track specification file [/home/administrator/.rally/benchmarks/tracks/default/pmc/track.json].  
> 2023-06-10 13:44:54,89 ActorAddr-(T|:38863)/PID:120075 esrally.track.loader INFO Final rendered track for '/home/administrator/.rally/benchmarks/tracks/default/pmc/track.json' has been written to '/tmp/tmp0f5613wg.json'.  
> 2023-06-10 13:44:54,97 ActorAddr-(T|:38863)/PID:120075 esrally.track.loader INFO Loading template [definition for index pmc in index.json].  
> 2023-06-10 13:44:54,102 ActorAddr-(T|:38863)/PID:120075 esrally.utils.modules INFO Loading component [pmc] from [/home/administrator/.rally/benchmarks/tracks/default/pmc]  
> 2023-06-10 13:44:54,106 ActorAddr-(T|:38863)/PID:120075 esrally.actor INFO Preparing track [pmc]  
> 2023-06-10 13:44:54,106 ActorAddr-(T|:38863)/PID:120075 esrally.actor INFO Reloading track [pmc] to ensure plugins are up-to-date.  
> 2023-06-10 13:44:54,971 ActorAddr-(T|:38863)/PID:120075 esrally.track.loader INFO Reading track specification file [/home/administrator/.rally/benchmarks/tracks/default/pmc/track.json].  
> 2023-06-10 13:44:55,6 ActorAddr-(T|:38863)/PID:120075 esrally.track.loader INFO Final rendered track for '/home/administrator/.rally/benchmarks/tracks/default/pmc/track.json' has been written to '/tmp/tmpw11nztxy.json'.  
> 2023-06-10 13:44:55,14 ActorAddr-(T|:38863)/PID:120075 esrally.track.loader INFO Loading template [definition for index pmc in index.json].  
> 2023-06-10 13:44:55,18 ActorAddr-(T|:38863)/PID:120075 esrally.utils.modules INFO Loading component [pmc] from [/home/administrator/.rally/benchmarks/tracks/default/pmc]  
> 2023-06-10 13:44:55,972 ActorAddr-(T|:35919)/PID:120103 esrally.utils.modules INFO Loading component [pmc] from [/home/administrator/.rally/benchmarks/tracks/default/pmc]  
> 2023-06-10 13:44:55,972 ActorAddr-(T|:45841)/PID:120106 esrally.utils.modules INFO Loading component [pmc] from [/home/administrator/.rally/benchmarks/tracks/default/pmc]  
> 2023-06-10 13:44:55,847 ActorAddr-(T|:38863)/PID:120075 esrally.utils.modules INFO Loading component [pmc] from [/home/administrator/.rally/benchmarks/tracks/default/pmc]  
> 2023-06-10 13:44:55,973 ActorAddr-(T|:44811)/PID:120104 esrally.utils.modules INFO Loading component [pmc] from [/home/administrator/.rally/benchmarks/tracks/default/pmc]  
> 2023-06-10 13:44:55,974 ActorAddr-(T|:37641)/PID:120105 esrally.utils.modules INFO Loading component [pmc] from [/home/administrator/.rally/benchmarks/tracks/default/pmc]  
> 2023-06-10 13:44:55,976 ActorAddr-(T|:42497)/PID:120107 esrally.utils.modules INFO Loading component [pmc] from [/home/administrator/.rally/benchmarks/tracks/default/pmc]  
> 2023-06-10 13:45:06,72 ActorAddr-(T|:41271)/PID:120143 esrally.utils.modules INFO Loading component [pmc] from [/home/administrator/.rally/benchmarks/tracks/default/pmc]  
> 2023-06-10 13:45:06,72 ActorAddr-(T|:33821)/PID:120147 esrally.utils.modules INFO Loading component [pmc] from [/home/administrator/.rally/benchmarks/tracks/default/pmc]  
> 2023-06-10 13:45:06,72 ActorAddr-(T|:38313)/PID:120145 esrally.utils.modules INFO Loading component [pmc] from [/home/administrator/.rally/benchmarks/tracks/default/pmc]  
> 2023-06-10 13:45:06,73 ActorAddr-(T|:39089)/PID:120148 esrally.utils.modules INFO Loading component [pmc] from [/home/administrator/.rally/benchmarks/tracks/default/pmc]  
> 2023-06-10 13:45:06,73 ActorAddr-(T|:33045)/PID:120150 esrally.utils.modules INFO Loading component [pmc] from [/home/administrator/.rally/benchmarks/tracks/default/pmc]  
> 2023-06-10 13:45:06,74 ActorAddr-(T|:39993)/PID:120149 esrally.utils.modules INFO Loading component [pmc] from [/home/administrator/.rally/benchmarks/tracks/default/pmc]  
> 2023-06-10 13:45:06,75 ActorAddr-(T|:35919)/PID:120103 esrally.track.loader INFO Resolved data root directory for document corpus [pmc] in track [pmc] to [['/home/administrator/.rally/benchmarks/data/pmc']].  
> **2023-06-10 13:45:06,76 ActorAddr-(T|:35919)/PID:120103 esrally.track.loader INFO Downloading data from [[https://rally-tracks.elastic.co/pmc/documents.json.bz2](https://rally-tracks.elastic.co/pmc/documents.json.bz2)] (5657 MB) to [/home/administrator/.rally/benchmarks/data/pmc/documents.json.bz2].**  
> **\> 2023-06-10 13:45:11,80 ActorAddr-(T|:35919)/PID:120103 esrally.driver.driver ERROR Worker failed. Notifying parent...**  
> Traceback (most recent call last):
> 
> File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen  
> httplib\_response = self.\_make\_request(
> 
> File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/connectionpool.py", line 386, in \_make\_request  
> self.\_validate\_conn(conn)
> 
> File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1040, in \_validate\_conn  
> conn.connect()
> 
> File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/connection.py", line 414, in connect  
> self.sock = ssl\_wrap\_socket(
> 
> File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/util/ssl\_.py", line 449, in ssl\_wrap\_socket  
> ssl\_sock = \_ssl\_wrap\_socket\_impl(
> 
> File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/util/ssl\_.py", line 493, in \_ssl\_wrap\_socket\_impl  
> return ssl\_context.wrap\_socket(sock, server\_hostname=server\_hostname)
> 
> File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/ssl.py", line 500, in wrap\_socket  
> return self.sslsocket\_class.\_create(
> 
> File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/ssl.py", line 1040, in \_create  
> self.do\_handshake()
> 
> File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/ssl.py", line 1309, in do\_handshake  
> self.\_sslobj.do\_handshake()
> 
> ssl.SSLCertVerificationError: [SSL: CERTIFICATE\_VERIFY\_FAILED] certificate verify failed: self signed certificate in certificate chain (\_ssl.c:1131)
> 
> During handling of the above exception, another exception occurred:
> 
> Traceback (most recent call last):
> 
> File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/concurrent/futures/thread.py", line 57, in run  
> result = self.fn(\*self.args, \*\*self.kwargs)
> 
> File "/home/administrator/.local/lib/python3.8/site-packages/esrally/track/loader.py", line 457, in prepare\_docs  
> preparator.prepare\_document\_set(document\_set, data\_root[0])
> 
> File "/home/administrator/.local/lib/python3.8/site-packages/esrally/track/loader.py", line 620, in prepare\_document\_set  
> self.downloader.download(document\_set.base\_url, target\_path, expected\_size)
> 
> File "/home/administrator/.local/lib/python3.8/site-packages/esrally/track/loader.py", line 528, in download  
> net.download(data\_url, target\_path, size\_in\_bytes, progress\_indicator=progress)
> 
> File "/home/administrator/.local/lib/python3.8/site-packages/esrally/utils/net.py", line 260, in download  
> expected\_size\_in\_bytes = download\_http(url, tmp\_data\_set\_path, expected\_size\_in\_bytes, progress\_indicator)
> 
> File "/home/administrator/.local/lib/python3.8/site-packages/esrally/utils/net.py", line 217, in download\_http  
> return \_download\_http(url, local\_path, expected\_size\_in\_bytes, progress\_indicator)
> 
> File "/home/administrator/.local/lib/python3.8/site-packages/esrally/utils/net.py", line 190, in \_download\_http  
> with \_request(
> 
> File "/home/administrator/.local/lib/python3.8/site-packages/esrally/utils/net.py", line 287, in \_request  
> return manager.request(method, url, \*\*kwargs)
> 
> File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/request.py", line 74, in request  
> return self.request\_encode\_url(
> 
> File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/request.py", line 96, in request\_encode\_url  
> return self.urlopen(method, url, \*\*extra\_kw)
> 
> File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/poolmanager.py", line 376, in urlopen  
> response = conn.urlopen(method, u.request\_uri, \*\*kw)
> 
> File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/connectionpool.py", line 813, in urlopen  
> return self.urlopen(
> 
> File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/connectionpool.py", line 813, in urlopen  
> return self.urlopen(
> 
> File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/connectionpool.py", line 813, in urlopen  
> return self.urlopen(
> 
> [Previous line repeated 7 more times]
> 
> File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/connectionpool.py", line 785, in urlopen  
> retries = retries.increment(
> 
> File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/util/retry.py", line 592, in increment  
> raise MaxRetryError(\_pool, url, error or ResponseError(cause))
> 
> urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='[rally-tracks.elastic.co](http://rally-tracks.elastic.co)', port=443): Max retries exceeded with url: /pmc/documents.json.bz2 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE\_VERIFY\_FAILED] certificate verify failed: self signed certificate in certificate chain (\_ssl.c:1131)')))
> 
> 2023-06-10 13:45:11,91 ActorAddr-(T|:34865)/PID:120074 esrally.actor ERROR Main driver received a fatal exception from a load generator. Shutting down.  
> 2023-06-10 13:45:11,91 ActorAddr-(T|:34865)/PID:120074 esrally.metrics INFO Closing metrics store.  
> 2023-06-10 13:45:11,96 -not-actor-/PID:120043 esrally.racecontrol ERROR A benchmark failure has occurred  
> 2023-06-10 13:45:11,97 -not-actor-/PID:120043 esrally.racecontrol INFO Telling benchmark actor to exit.  
> 2023-06-10 13:45:11,94 ActorAddr-(T|:39433)/PID:120054 esrally.actor INFO Received a benchmark failure from [ActorAddr-(T|:34865)] and will forward it now.  
> 2023-06-10 13:45:11,100 ActorAddr-(T|:34865)/PID:120074 esrally.actor INFO Main driver received ActorExitRequest and will terminate all load generators.  
> 2023-06-10 13:45:14,101 -not-actor-/PID:120043 esrally.rally INFO Attempting to shutdown internal actor system.  
> 2023-06-10 13:45:14,104 -not-actor-/PID:120053 root INFO ActorSystem Logging Shutdown  
> 2023-06-10 13:45:14,126 -not-actor-/PID:120052 root INFO ---- Actor System shutdown  
> 2023-06-10 13:45:14,126 -not-actor-/PID:120043 esrally.rally INFO Actor system is still running. Waiting...  
> 2023-06-10 13:45:15,128 -not-actor-/PID:120043 esrally.rally INFO Shutdown completed.  
> 2023-06-10 13:45:15,128 -not-actor-/PID:120043 esrally.rally ERROR Cannot run subcommand [race].  
> Traceback (most recent call last):  
> File "/home/administrator/.local/lib/python3.8/site-packages/esrally/rally.py", line 1171, in dispatch\_sub\_command  
> race(cfg, args.kill\_running\_processes)  
> File "/home/administrator/.local/lib/python3.8/site-packages/esrally/rally.py", line 919, in race  
> with\_actor\_system(racecontrol.run, cfg)  
> File "/home/administrator/.local/lib/python3.8/site-packages/esrally/rally.py", line 949, in with\_actor\_system  
> runnable(cfg)  
> File "/home/administrator/.local/lib/python3.8/site-packages/esrally/racecontrol.py", line 372, in run  
> raise e  
> File "/home/administrator/.local/lib/python3.8/site-packages/esrally/racecontrol.py", line 369, in run  
> pipeline(cfg)  
> File "/home/administrator/.local/lib/python3.8/site-packages/esrally/racecontrol.py", line 71, in **call**  
> self.target(cfg)  
> File "/home/administrator/.local/lib/python3.8/site-packages/esrally/racecontrol.py", line 308, in benchmark\_only  
> return race(cfg, external=True)  
> File "/home/administrator/.local/lib/python3.8/site-packages/esrally/racecontrol.py", line 266, in race  
> raise exceptions.RallyError(result.message, result.cause)  
> esrally.exceptions.RallyError: Error in task executor

Can someone guide me here?

---

<div class="post-metadata">

**Author:** ![Quentin\_Pradet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/quentin_pradet/32/94192_2.png) [@Quentin\_Pradet](https://discuss.elastic.co/u/Quentin_Pradet)\
**Post date:** [June 12, 2023, 6:48am UTC](https://discuss.elastic.co/t/max-retries-exceeded-with-url-pmc-documents-json-bz2-sslcertverificationerror-ssl-certificate-verify-failed/335708/2 "2023-06-12T06:48:55Z")

</div>

Hello, and thanks for your interest in Rally.

The issue here is that you're telling Rally to use `/home/administrator/ca.crt` for certificate authorities and nothing else. But [https://rally-tracks.elastic.co/](https://rally-tracks.elastic.co/) needs a _different_ certificate authority, one of the defaults one from the [certifi · PyPI](https://pypi.org/project/certifi/) package.

I'm seeing two options here:

- Since you're on Ubuntu, follow [Installing a root CA certificate in the trust store | Ubuntu](https://ubuntu.com/server/docs/security-trust-store) to add you local ca.crt file to the global store, and then configure `ca_certs` to `/etc/ssl/certs/ca-certificates.crt`
- Download tracks separately using the offline mode: [Offline Usage - Rally 2.7.1 documentation](https://esrally.readthedocs.io/en/stable/offline.html)

---

<div class="post-metadata">

**Author:** ![Quentin\_Pradet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/quentin_pradet/32/94192_2.png) [@Quentin\_Pradet](https://discuss.elastic.co/u/Quentin_Pradet)\
**Post date:** [June 12, 2023, 12:49pm UTC](https://discuss.elastic.co/t/max-retries-exceeded-with-url-pmc-documents-json-bz2-sslcertverificationerror-ssl-certificate-verify-failed/335708/3 "2023-06-12T12:49:19Z")

</div>

Actually, downloading tracks does not use the `--client-options` configuration, so this is puzzling. Can you please run the following script? This is what Rally does under the hood. This will help diagnose the issue.

```python
import certifi
import urllib3
import esrally

print(esrally. __version__ , urllib3. __version__ , certifi. __version__ )
http = urllib3.PoolManager(cert_reqs="CERT_REQUIRED", ca_certs=certifi.where())
http.request("GET", "https://rally-tracks.elastic.co")
response = http.request("GET", "https://rally-tracks.elastic.co")
assert response.status == 200

```

---

<div class="post-metadata">

**Author:** ![Swathi\_Kakumanu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/swathi_kakumanu/32/117764_2.png) [@Swathi\_Kakumanu](https://discuss.elastic.co/u/Swathi_Kakumanu)\
**Post date:** [June 13, 2023, 1:45pm UTC](https://discuss.elastic.co/t/max-retries-exceeded-with-url-pmc-documents-json-bz2-sslcertverificationerror-ssl-certificate-verify-failed/335708/4 "2023-06-13T13:45:11Z")

</div>

Sure, this the output:

`administrator@administrator:~$ python3 rally_http_test.py`

> 2.7.1 1.26.9 2023.05.07  
> Traceback (most recent call last):  
> File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/connectionpool.py", line 703, in urlopen  
> httplib\_response = self.\_make\_request(  
> File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/connectionpool.py", line 386, in \_make\_request  
> self.\_validate\_conn(conn)  
> File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/connectionpool.py", line 1040, in _validate\_conn  
> conn.connect()  
> File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/connection.py", line 414, in connect  
> self.sock = ssl\_wrap\_socket(  
> File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 449, in ssl\_wrap\_socket  
> ssl\_sock = _ssl\_wrap\_socket\_impl(  
> File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/util/ssl_.py", line 493, in \_ssl\_wrap\_socket\_impl  
> return ssl\_context.wrap\_socket(sock, server\_hostname=server\_hostname)  
> File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/ssl.py", line 500, in wrap\_socket  
> return self.sslsocket\_class.\_create(  
> File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/ssl.py", line 1040, in \_create  
> self.do\_handshake()  
> File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/ssl.py", line 1309, in do\_handshake  
> self.\_sslobj.do\_handshake()  
> ssl.SSLCertVerificationError: [SSL: CERTIFICATE\_VERIFY\_FAILED] certificate verify failed: self signed certificate in certificate chain (\_ssl.c:1131)
> 
> During handling of the above exception, another exception occurred:
> 
> Traceback (most recent call last):  
> File "rally\_http\_test.py", line 8, in   
> http.request("GET", "[https://rally-tracks.elastic.co](https://rally-tracks.elastic.co)")  
> File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/request.py", line 74, in request  
> return self.request\_encode\_url(  
> File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/request.py", line 96, in request\_encode\_url  
> return self.urlopen(method, url, \*\*extra\_kw)  
> File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/poolmanager.py", line 376, in urlopen  
> response = conn.urlopen(method, u.request\_uri, \*\*kw)  
> File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/connectionpool.py", line 813, in urlopen  
> return self.urlopen(  
> File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/connectionpool.py", line 813, in urlopen  
> return self.urlopen(  
> File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/connectionpool.py", line 813, in urlopen  
> return self.urlopen(  
> File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/connectionpool.py", line 785, in urlopen  
> retries = retries.increment(  
> File "/home/administrator/.pyenv/versions/3.8.13/lib/python3.8/site-packages/urllib3/util/retry.py", line 592, in increment  
> raise MaxRetryError(\_pool, url, error or ResponseError(cause))  
> urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='[rally-tracks.elastic.co](http://rally-tracks.elastic.co)', port=443): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE\_VERIFY\_FAILED] certificate verify failed: self signed certificate in certificate chain (\_ssl.c:1131)')))

---

<div class="post-metadata">

**Author:** ![Swathi\_Kakumanu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/swathi_kakumanu/32/117764_2.png) [@Swathi\_Kakumanu](https://discuss.elastic.co/u/Swathi_Kakumanu)\
**Post date:** [June 13, 2023, 2:26pm UTC](https://discuss.elastic.co/t/max-retries-exceeded-with-url-pmc-documents-json-bz2-sslcertverificationerror-ssl-certificate-verify-failed/335708/5 "2023-06-13T14:26:35Z")

</div>

What I have observed is:

`certifi.where()` is showing as ` /home/administrator/.local/lib/python3.8/site-packages/certifi/cacert.pem` and my ca root certificate is in `/home/administrator/`

from your previous instructions, I performed the below steps:

1. Copied these two files (ca.crt and cacert.pem) to `/usr/local/share/ca-certificates` and ran the `sudo update-ca-certificates`
2. Changed the ca\_certs value in the test python script from `ca_certs=certifi.where()` to `ca_certs='/etc/ssl/certs/ca-certificates.crt'`
3. Executed the python file, this time I didn't get any error. \> administrator@administrator:~$ python3 rally\_http\_test.py

> 2.7.1 1.26.9 2023.05.07

However, If if I configure this certificate in the esrally command line, then I am still getting the certificate error:

`esrally race --pipeline benchmark-only --track=geopoint --challenge append-no-conflicts-index-only --target-hosts=192.168.105.116:9200 --track-params="bulk_indexing_clients:10" --client-options="use_ssl:true,verify_certs:true,ca_certs:'/etc/ssl/certs/ca-certificates.crt',basic_auth_user:'elastic',basic_auth_password:'xxxxx'"`

> [ERROR] Cannot race. Error in task executor  
> HTTPSConnectionPool(host='[rally-tracks.elastic.co](http://rally-tracks.elastic.co)', port=443): Max retries exceeded with url: /geopoint/documents.json.bz2 (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE\_VERIFY\_FAILED] certificate verify failed: self signed certificate in certificate chain (\_ssl.c:1131)')))

Is there any other way to provide this value to the esrally command? is this the right direction?

---

<div class="post-metadata">

**Author:** ![Quentin\_Pradet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/quentin_pradet/32/94192_2.png) [@Quentin\_Pradet](https://discuss.elastic.co/u/Quentin_Pradet)\
**Post date:** [June 15, 2023, 8:53am UTC](https://discuss.elastic.co/t/max-retries-exceeded-with-url-pmc-documents-json-bz2-sslcertverificationerror-ssl-certificate-verify-failed/335708/6 "2023-06-15T08:53:00Z")

</div>

Yes, this is the correct direction. It looks like your Rally VM is behind a MITM SSL proxy which means you need a different SSL certificate to connect to the Internet. Maybe this is this ca.crt file, maybe this is configured already in the Ubuntu system certificates, it’s hard to tell from your experiment.

Unfortunately Rally does not allow configuring the certificate authorities used to download tracks: [rally/esrally/utils/net.py at 675f1bdbe1fb96f07fefd947a1cd316525544b3a · elastic/rally · GitHub](https://github.com/elastic/rally/blob/675f1bdbe1fb96f07fefd947a1cd316525544b3a/esrally/utils/net.py#L52)

But again you can use offline mode by downloading the tracks separately and the sending them to the Rally VM: [Offline Usage - Rally 2.8.0 documentation](https://esrally.readthedocs.io/en/stable/offline.html)

(Thanks @sethmlarson for the help here)

---

<div class="post-metadata">

**Author:** ![Swathi\_Kakumanu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/swathi_kakumanu/32/117764_2.png) [@Swathi\_Kakumanu](https://discuss.elastic.co/u/Swathi_Kakumanu)\
**Post date:** [June 19, 2023, 5:04am UTC](https://discuss.elastic.co/t/max-retries-exceeded-with-url-pmc-documents-json-bz2-sslcertverificationerror-ssl-certificate-verify-failed/335708/7 "2023-06-19T05:04:40Z")

</div>

Thanks for the guidance Quentin\_Pradet

---

<div class="post-metadata">

**Author:** ![Swathi\_Kakumanu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/swathi_kakumanu/32/117764_2.png) [@Swathi\_Kakumanu](https://discuss.elastic.co/u/Swathi_Kakumanu)\
**Post date:** [July 5, 2023, 3:51pm UTC](https://discuss.elastic.co/t/max-retries-exceeded-with-url-pmc-documents-json-bz2-sslcertverificationerror-ssl-certificate-verify-failed/335708/8 "2023-07-05T15:51:39Z")

</div>

Hi @Quentin_Pradet , as a workaround for online execution, I did the below changes.

1st Step : Updating the cacerts

```auto
sudo cp ca.crt /usr/local/share/ca-certificates
sudo cp /home/administrator/.local/lib/python3.8/site-packages/certifi/cacert.pem /usr/local/share/ca-certificates
sudo update-ca-certificates

```

which will update the '/etc/ssl/certs/ca-certificates.crt'

2nd Step: Replacing the default certifi file with the updated one:

- Verify certifi.where() path, in my case it is /home/administrator/.local/lib/python3.8/site-packages/certifi/cacert.pem
- Take a backup of this file just for reference
- Copy the file /etc/ssl/certs/ca-certificates.crt as cacert.pem and replace in the path /home/administrator/.local/lib/python3.8/site-packages/certifi/

> So that certifi will find this certificate which has external access now. Basically replacing the default file.

With these steps I was able to execute online mode.

Just sharing it here so that it can help others.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 2, 2023, 3:52pm UTC](https://discuss.elastic.co/t/max-retries-exceeded-with-url-pmc-documents-json-bz2-sslcertverificationerror-ssl-certificate-verify-failed/335708/9 "2023-08-02T15:52:00Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.