# Esrally connection aborted **URL:** \ **Category:** Elasticsearch\ **Tags:** rally\ **Created:** [February 29, 2024, 11:51am UTC](https://discuss.elastic.co/t/esrally-connection-aborted/354440 "2024-02-29T11:51:45Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![uttamkrpanda](https://avatars.discourse-cdn.com/v4/letter/u/3da27b/32.png) [@uttamkrpanda](https://discuss.elastic.co/u/uttamkrpanda)\ **Post date:** [February 29, 2024, 11:51am UTC](https://discuss.elastic.co/t/esrally-connection-aborted/354440/1 "2024-02-29T11:51:45Z")
I am trying to run the esrally in local , however it's getting connection aborted . The curl to the elasticsearch host is working fine but not via esrally . Any help here please ? ```auto esrally race --track=pmc --target-hosts=localhost:9200 --pipeline=benchmark-only --client-options="basic_auth_user:'elastic',basic_auth_password:'04CA3sd8FN7lYkfe6l27CW03'" ``` ```auto [INFO] Race id is [85121376-cadf-45a5-bb83-4e13ecd6831c] [ERROR] Cannot race. Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/esrally/actor.py", line 92, in guard return f(self, msg, sender) File "/usr/local/lib/python3.9/site-packages/esrally/racecontrol.py", line 111, in receiveMsg_Setup self.coordinator.setup(sources=msg.sources) File "/usr/local/lib/python3.9/site-packages/esrally/racecontrol.py", line 194, in setup ) = client.factory.cluster_distribution_version(hosts, client_options) File "/usr/local/lib/python3.9/site-packages/esrally/client/factory.py", line 349, in cluster_distribution_version version = es.info()["version"] File "/usr/local/lib/python3.9/site-packages/elasticsearch/_sync/client/utils.py", line 414, in wrapped return api(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/elasticsearch/_sync/client/ __init__.py", line 2296, in info return self.perform_request( # type: ignore[return-value] File "/usr/local/lib/python3.9/site-packages/esrally/client/synchronous.py", line 168, in perform_request info = self.transport.perform_request(method="GET", target="/", headers=request_headers) File "/usr/local/lib/python3.9/site-packages/elastic_transport/_transport.py", line 328, in perform_request meta, raw_data = node.perform_request( File "/usr/local/lib/python3.9/site-packages/elastic_transport/_node/_http_urllib3.py", line 197, in perform_request raise err from None elastic_transport.ConnectionError: Connection error caused by: ConnectionError(Connection error caused by: ProtocolError(('Connection aborted.', RemoteDisconnected('Remote end closed connection without response')))) ``` I am using esrally : 2.10.0 and python : 3.9.18 ---
**Author:** ![gbanasiak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gbanasiak/32/121454_2.png) [@gbanasiak](https://discuss.elastic.co/u/gbanasiak)\ **Post date:** [February 29, 2024, 12:08pm UTC](https://discuss.elastic.co/t/esrally-connection-aborted/354440/2 "2024-02-29T12:08:41Z")
Rally does not use TLS/SSL by default which might be causing that. Can you try adding `use_ssl:true,ca_certs:''` to client options and comparing results? You can also try with `verify-certs:false` instead of `ca_certs` file path, as a quick test. If this doesn't help, please share curl invocation that works. ---
**Author:** ![uttamkrpanda](https://avatars.discourse-cdn.com/v4/letter/u/3da27b/32.png) [@uttamkrpanda](https://discuss.elastic.co/u/uttamkrpanda)\ **Post date:** [February 29, 2024, 2:05pm UTC](https://discuss.elastic.co/t/esrally-connection-aborted/354440/3 "2024-02-29T14:05:55Z")
Hi @gbanasiak , if i make verify-certs: false getting below error ```auto esrally race --track=pmc --target-hosts=localhost:9200 --pipeline=benchmark-only --client-options="use_ssl:true,verify_certs:false,basic_auth_user:'elastic',basic_auth_password:'04CA3sd8FN7lYkfe6l27CW03'" ``` ```auto INFO] Race id is [1e93e917-6889-403e-bc15-2b5a94898093] [ERROR] Cannot race. Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/esrally/actor.py", line 92, in guard return f(self, msg, sender) File "/usr/local/lib/python3.9/site-packages/esrally/racecontrol.py", line 111, in receiveMsg_Setup self.coordinator.setup(sources=msg.sources) File "/usr/local/lib/python3.9/site-packages/esrally/racecontrol.py", line 235, in setup self.metrics_store = metrics.metrics_store( File "/usr/local/lib/python3.9/site-packages/esrally/metrics.py", line 358, in metrics_store store = cls(cfg=cfg, meta_info=meta_info) File "/usr/local/lib/python3.9/site-packages/esrally/metrics.py", line 895, in __init__ self._client = client_factory_class(cfg).create() File "/usr/local/lib/python3.9/site-packages/esrally/metrics.py", line 267, in __init__ distribution_flavor, distribution_version, _, _ = client.cluster_distribution_version( File "/usr/local/lib/python3.9/site-packages/esrally/client/factory.py", line 349, in cluster_distribution_version version = es.info()["version"] File "/usr/local/lib/python3.9/site-packages/elasticsearch/_sync/client/utils.py", line 414, in wrapped return api(*args, **kwargs) File "/usr/local/lib/python3.9/site-packages/elasticsearch/_sync/client/ __init__.py", line 2296, in info return self.perform_request( # type: ignore[return-value] File "/usr/local/lib/python3.9/site-packages/esrally/client/synchronous.py", line 168, in perform_request info = self.transport.perform_request(method="GET", target="/", headers=request_headers) File "/usr/local/lib/python3.9/site-packages/elastic_transport/_transport.py", line 328, in perform_request meta, raw_data = node.perform_request( File "/usr/local/lib/python3.9/site-packages/elastic_transport/_node/_http_urllib3.py", line 197, in perform_request raise err from None elastic_transport.TlsError: TLS error caused by: TlsError(TLS error caused by: SSLError([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129))) ``` if i pass the pem file , getting file not found error ```auto esrally race --track=pmc --target-hosts=localhost:9200 --pipeline=benchmark-only --client-options="use_ssl:true,verify_certs:true, ca_certs:'/home/cloud-user/key.pem,basic_auth_user:'elastic',basic_auth_password:'04CA3sd8FN7lYkfe6l27CW03'" ``` ```auto [INFO] Race id is [ec2ef8eb-9230-49f5-b014-62a9b2728c63] [ERROR] Cannot race. Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/esrally/actor.py", line 92, in guard return f(self, msg, sender) File "/usr/local/lib/python3.9/site-packages/esrally/racecontrol.py", line 111, in receiveMsg_Setup self.coordinator.setup(sources=msg.sources) File "/usr/local/lib/python3.9/site-packages/esrally/racecontrol.py", line 194, in setup ) = client.factory.cluster_distribution_version(hosts, client_options) File "/usr/local/lib/python3.9/site-packages/esrally/client/factory.py", line 345, in cluster_distribution_version es = client_factory(hosts, client_options).create() File "/usr/local/lib/python3.9/site-packages/esrally/client/factory.py", line 66, in __init__ self.ssl_context = ssl.create_default_context( File "/usr/lib64/python3.9/ssl.py", line 746, in create_default_context context.load_verify_locations(cafile, capath, cadata) FileNotFoundError: [Errno 2] No such file or directory ``` below is the curl response ```auto [root@ipXX .rally]# curl -s -u "elastic:04CA3sd8FN7lYkfe6l27CW03" -k "https://localhost:9200/_cat/indices?h=index" .geoip_databases .security-7 test.02 test.01 ``` ---
**Author:** ![gbanasiak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gbanasiak/32/121454_2.png) [@gbanasiak](https://discuss.elastic.co/u/gbanasiak)\ **Post date:** [February 29, 2024, 2:46pm UTC](https://discuss.elastic.co/t/esrally-connection-aborted/354440/4 "2024-02-29T14:46:29Z")
With your first command you've run into TLS verification failure for your Elasticsearch metric store which should be a different cluster than the one you're testing. Please try adding the following in `[reporting]` section in `~/.rally/rally.ini` file (see [doc](https://esrally.readthedocs.io/en/stable/configuration.html?highlight=datastore#reporting)): ```auto datastore.ssl.verification_mode = none ``` Or disable Elasticsearch metric store completely for now. Your second command has incorrect quotes for `ca_certs` client option I think. ---
**Author:** ![uttamkrpanda](https://avatars.discourse-cdn.com/v4/letter/u/3da27b/32.png) [@uttamkrpanda](https://discuss.elastic.co/u/uttamkrpanda)\ **Post date:** [February 29, 2024, 5:18pm UTC](https://discuss.elastic.co/t/esrally-connection-aborted/354440/5 "2024-02-29T17:18:27Z")
hi @gbanasiak , yeah for second command i am getting like this now . ```auto esrally race --track=pmc --target-hosts=localhost:9200 --pipeline=benchmark-only --client-options="use_ssl:true,verify_certs:true,ca_certs:'/home/cloud-user/.rally/key.pem'" ________ / __\____ _/ / /_ __ / /_/ / __ `/ / / / / / / _, _/ /_/ / / / /_/ / /_/ |_|\ __,_/_/_/\__ , / / ____ / [INFO] Race id is [a786ff9a-9b92-47e3-9162-b74ad9cf0200] [ERROR] Cannot race. Traceback (most recent call last): File "/usr/local/lib/python3.9/site-packages/esrally/actor.py", line 92, in guard return f(self, msg, sender) File "/usr/local/lib/python3.9/site-packages/esrally/racecontrol.py", line 111, in receiveMsg_Setup self.coordinator.setup(sources=msg.sources) File "/usr/local/lib/python3.9/site-packages/esrally/racecontrol.py", line 194, in setup ) = client.factory.cluster_distribution_version(hosts, client_options) File "/usr/local/lib/python3.9/site-packages/esrally/client/factory.py", line 345, in cluster_distribution_version es = client_factory(hosts, client_options).create() File "/usr/local/lib/python3.9/site-packages/esrally/client/factory.py", line 66, in __init__ self.ssl_context = ssl.create_default_context( File "/usr/lib64/python3.9/ssl.py", line 746, in create_default_context context.load_verify_locations(cafile, capath, cadata) ssl.SSLError: [X509: NO_CERTIFICATE_OR_CRL_FOUND] no certificate or crl found (_ssl.c:4293) ``` ---
**Author:** ![uttamkrpanda](https://avatars.discourse-cdn.com/v4/letter/u/3da27b/32.png) [@uttamkrpanda](https://discuss.elastic.co/u/uttamkrpanda)\ **Post date:** [February 29, 2024, 5:27pm UTC](https://discuss.elastic.co/t/esrally-connection-aborted/354440/6 "2024-02-29T17:27:40Z")
The elasticsearch is running in kubernetes cluster . I did port forwarding on 9200 , where curl is working perfectly fine not the esrally . bdw how to disable Elasticsearch metric store completely ? any steps we have ? ---
**Author:** ![gbanasiak](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gbanasiak/32/121454_2.png) [@gbanasiak](https://discuss.elastic.co/u/gbanasiak)\ **Post date:** [March 1, 2024, 8:09am UTC](https://discuss.elastic.co/t/esrally-connection-aborted/354440/7 "2024-03-01T08:09:59Z")
> [@uttamkrpanda](#): > > `ssl.SSLError: [X509: NO_CERTIFICATE_OR_CRL_FOUND] no certificate or crl found (_ssl.c:4293)` This suggests your `/home/cloud-user/.rally/key.pem` file has incorrect content. If you can make `curl` work without `-k` option providing certificates file with `--cacert` option, the same file should also work fine with Rally. > bdw how to disable Elasticsearch metric store completely ? any steps we have ? Here is the default metric store configuration from `rally.ini`. It uses in-memory metric store as opposed to Elasticsearch metric store: ```auto [reporting] datastore.type = in-memory datastore.host = datastore.port = datastore.secure = False datastore.user = datastore.password = ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 29, 2024, 8:10am UTC](https://discuss.elastic.co/t/esrally-connection-aborted/354440/8 "2024-03-29T08:10:36Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.