Esrally connection aborted

uttamkrpanda · February 29, 2024, 11:51am

I am trying to run the esrally in local , however it's getting connection aborted .
The curl to the elasticsearch host is working fine but not via esrally . Any help here please ?

 esrally race --track=pmc --target-hosts=localhost:9200 --pipeline=benchmark-only --client-options="basic_auth_user:'elastic',basic_auth_password:'04CA3sd8FN7lYkfe6l27CW03'"


[INFO] Race id is [85121376-cadf-45a5-bb83-4e13ecd6831c]
[ERROR] Cannot race. Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/esrally/actor.py", line 92, in guard
    return f(self, msg, sender)
  File "/usr/local/lib/python3.9/site-packages/esrally/racecontrol.py", line 111, in receiveMsg_Setup
    self.coordinator.setup(sources=msg.sources)
  File "/usr/local/lib/python3.9/site-packages/esrally/racecontrol.py", line 194, in setup
    ) = client.factory.cluster_distribution_version(hosts, client_options)
  File "/usr/local/lib/python3.9/site-packages/esrally/client/factory.py", line 349, in cluster_distribution_version
    version = es.info()["version"]
  File "/usr/local/lib/python3.9/site-packages/elasticsearch/_sync/client/utils.py", line 414, in wrapped
    return api(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/elasticsearch/_sync/client/__init__.py", line 2296, in info
    return self.perform_request(  # type: ignore[return-value]
  File "/usr/local/lib/python3.9/site-packages/esrally/client/synchronous.py", line 168, in perform_request
    info = self.transport.perform_request(method="GET", target="/", headers=request_headers)
  File "/usr/local/lib/python3.9/site-packages/elastic_transport/_transport.py", line 328, in perform_request
    meta, raw_data = node.perform_request(
  File "/usr/local/lib/python3.9/site-packages/elastic_transport/_node/_http_urllib3.py", line 197, in perform_request
    raise err from None
elastic_transport.ConnectionError: Connection error caused by: ConnectionError(Connection error caused by: ProtocolError(('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))))

I am using esrally : 2.10.0 and python : 3.9.18

gbanasiak · February 29, 2024, 12:08pm

Rally does not use TLS/SSL by default which might be causing that. Can you try adding use_ssl:true,ca_certs:'<path-to-pem-file>' to client options and comparing results? You can also try with verify-certs:false instead of ca_certs file path, as a quick test.

If this doesn't help, please share curl invocation that works.

uttamkrpanda · February 29, 2024, 2:05pm

Hi @gbanasiak ,

if i make verify-certs: false getting below error

esrally race --track=pmc --target-hosts=localhost:9200 --pipeline=benchmark-only --client-options="use_ssl:true,verify_certs:false,basic_auth_user:'elastic',basic_auth_password:'04CA3sd8FN7lYkfe6l27CW03'"

INFO] Race id is [1e93e917-6889-403e-bc15-2b5a94898093]
[ERROR] Cannot race. Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/esrally/actor.py", line 92, in guard
    return f(self, msg, sender)
  File "/usr/local/lib/python3.9/site-packages/esrally/racecontrol.py", line 111, in receiveMsg_Setup
    self.coordinator.setup(sources=msg.sources)
  File "/usr/local/lib/python3.9/site-packages/esrally/racecontrol.py", line 235, in setup
    self.metrics_store = metrics.metrics_store(
  File "/usr/local/lib/python3.9/site-packages/esrally/metrics.py", line 358, in metrics_store
    store = cls(cfg=cfg, meta_info=meta_info)
  File "/usr/local/lib/python3.9/site-packages/esrally/metrics.py", line 895, in __init__
    self._client = client_factory_class(cfg).create()
  File "/usr/local/lib/python3.9/site-packages/esrally/metrics.py", line 267, in __init__
    distribution_flavor, distribution_version, _, _ = client.cluster_distribution_version(
  File "/usr/local/lib/python3.9/site-packages/esrally/client/factory.py", line 349, in cluster_distribution_version
    version = es.info()["version"]
  File "/usr/local/lib/python3.9/site-packages/elasticsearch/_sync/client/utils.py", line 414, in wrapped
    return api(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/elasticsearch/_sync/client/__init__.py", line 2296, in info
    return self.perform_request(  # type: ignore[return-value]
  File "/usr/local/lib/python3.9/site-packages/esrally/client/synchronous.py", line 168, in perform_request
    info = self.transport.perform_request(method="GET", target="/", headers=request_headers)
  File "/usr/local/lib/python3.9/site-packages/elastic_transport/_transport.py", line 328, in perform_request
    meta, raw_data = node.perform_request(
  File "/usr/local/lib/python3.9/site-packages/elastic_transport/_node/_http_urllib3.py", line 197, in perform_request
    raise err from None
elastic_transport.TlsError: TLS error caused by: TlsError(TLS error caused by: SSLError([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)))

if i pass the pem file , getting file not found error

esrally race --track=pmc --target-hosts=localhost:9200 --pipeline=benchmark-only --client-options="use_ssl:true,verify_certs:true, ca_certs:'/home/cloud-user/key.pem,basic_auth_user:'elastic',basic_auth_password:'04CA3sd8FN7lYkfe6l27CW03'"

[INFO] Race id is [ec2ef8eb-9230-49f5-b014-62a9b2728c63]
[ERROR] Cannot race. Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/esrally/actor.py", line 92, in guard
    return f(self, msg, sender)
  File "/usr/local/lib/python3.9/site-packages/esrally/racecontrol.py", line 111, in receiveMsg_Setup
    self.coordinator.setup(sources=msg.sources)
  File "/usr/local/lib/python3.9/site-packages/esrally/racecontrol.py", line 194, in setup
    ) = client.factory.cluster_distribution_version(hosts, client_options)
  File "/usr/local/lib/python3.9/site-packages/esrally/client/factory.py", line 345, in cluster_distribution_version
    es = client_factory(hosts, client_options).create()
  File "/usr/local/lib/python3.9/site-packages/esrally/client/factory.py", line 66, in __init__
    self.ssl_context = ssl.create_default_context(
  File "/usr/lib64/python3.9/ssl.py", line 746, in create_default_context
    context.load_verify_locations(cafile, capath, cadata)
FileNotFoundError: [Errno 2] No such file or directory

below is the curl response

[root@ipXX .rally]# curl -s -u "elastic:04CA3sd8FN7lYkfe6l27CW03" -k "https://localhost:9200/_cat/indices?h=index"
.geoip_databases
.security-7
test.02
test.01

gbanasiak · February 29, 2024, 2:46pm

With your first command you've run into TLS verification failure for your Elasticsearch metric store which should be a different cluster than the one you're testing.

Please try adding the following in [reporting] section in ~/.rally/rally.ini file (see doc):

datastore.ssl.verification_mode = none

Or disable Elasticsearch metric store completely for now.

Your second command has incorrect quotes for ca_certs client option I think.

uttamkrpanda · February 29, 2024, 5:18pm

hi @gbanasiak ,

yeah for second command i am getting like this now .

esrally race --track=pmc --target-hosts=localhost:9200 --pipeline=benchmark-only --client-options="use_ssl:true,verify_certs:true,ca_certs:'/home/cloud-user/.rally/key.pem'"

    ____        ____
   / __ \____ _/ / /_  __
  / /_/ / __ `/ / / / / /
 / _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
                /____/

[INFO] Race id is [a786ff9a-9b92-47e3-9162-b74ad9cf0200]
[ERROR] Cannot race. Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/esrally/actor.py", line 92, in guard
    return f(self, msg, sender)
  File "/usr/local/lib/python3.9/site-packages/esrally/racecontrol.py", line 111, in receiveMsg_Setup
    self.coordinator.setup(sources=msg.sources)
  File "/usr/local/lib/python3.9/site-packages/esrally/racecontrol.py", line 194, in setup
    ) = client.factory.cluster_distribution_version(hosts, client_options)
  File "/usr/local/lib/python3.9/site-packages/esrally/client/factory.py", line 345, in cluster_distribution_version
    es = client_factory(hosts, client_options).create()
  File "/usr/local/lib/python3.9/site-packages/esrally/client/factory.py", line 66, in __init__
    self.ssl_context = ssl.create_default_context(
  File "/usr/lib64/python3.9/ssl.py", line 746, in create_default_context
    context.load_verify_locations(cafile, capath, cadata)
ssl.SSLError: [X509: NO_CERTIFICATE_OR_CRL_FOUND] no certificate or crl found (_ssl.c:4293)

uttamkrpanda · February 29, 2024, 5:27pm

The elasticsearch is running in kubernetes cluster . I did port forwarding on 9200 , where curl is working perfectly fine not the esrally .

bdw how to disable Elasticsearch metric store completely ? any steps we have ?

gbanasiak · March 1, 2024, 8:09am

This suggests your /home/cloud-user/.rally/key.pem file has incorrect content. If you can make curl work without -k option providing certificates file with --cacert option, the same file should also work fine with Rally.

bdw how to disable Elasticsearch metric store completely ? any steps we have ?

Here is the default metric store configuration from rally.ini. It uses in-memory metric store as opposed to Elasticsearch metric store:

[reporting]
datastore.type = in-memory
datastore.host =
datastore.port =
datastore.secure = False
datastore.user =
datastore.password =

system · March 29, 2024, 8:10am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Esrally "Could not connect to your Elasticsearch metrics store." Elasticsearch rally	4	635	December 2, 2021
Error Benchmarking AWS Managed Elasticsearch Cluster (version 7.10) using esrally Elasticsearch rally	11	279	June 9, 2024
Esrally failing with [ERROR] Cannot race. Error in race control (NotFoundError(404, 'Not Found', 'Not Found')) Elasticsearch rally	4	729	September 14, 2021
Transport 502 Error: Need Help Isolating Error Elasticsearch rally	3	998	July 11, 2022
Rally errors Elasticsearch rally	11	3225	July 5, 2017

Esrally connection aborted

Related topics