Esrally connection aborted

Elastic Stack Elasticsearch
1

I am trying to run the esrally in local , however it's getting connection aborted .
The curl to the elasticsearch host is working fine but not via esrally . Any help here please ?

 esrally race --track=pmc --target-hosts=localhost:9200 --pipeline=benchmark-only --client-options="basic_auth_user:'elastic',basic_auth_password:'04CA3sd8FN7lYkfe6l27CW03'"


[INFO] Race id is [85121376-cadf-45a5-bb83-4e13ecd6831c]
[ERROR] Cannot race. Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/esrally/actor.py", line 92, in guard
    return f(self, msg, sender)
  File "/usr/local/lib/python3.9/site-packages/esrally/racecontrol.py", line 111, in receiveMsg_Setup
    self.coordinator.setup(sources=msg.sources)
  File "/usr/local/lib/python3.9/site-packages/esrally/racecontrol.py", line 194, in setup
    ) = client.factory.cluster_distribution_version(hosts, client_options)
  File "/usr/local/lib/python3.9/site-packages/esrally/client/factory.py", line 349, in cluster_distribution_version
    version = es.info()["version"]
  File "/usr/local/lib/python3.9/site-packages/elasticsearch/_sync/client/utils.py", line 414, in wrapped
    return api(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/elasticsearch/_sync/client/__init__.py", line 2296, in info
    return self.perform_request(  # type: ignore[return-value]
  File "/usr/local/lib/python3.9/site-packages/esrally/client/synchronous.py", line 168, in perform_request
    info = self.transport.perform_request(method="GET", target="/", headers=request_headers)
  File "/usr/local/lib/python3.9/site-packages/elastic_transport/_transport.py", line 328, in perform_request
    meta, raw_data = node.perform_request(
  File "/usr/local/lib/python3.9/site-packages/elastic_transport/_node/_http_urllib3.py", line 197, in perform_request
    raise err from None
elastic_transport.ConnectionError: Connection error caused by: ConnectionError(Connection error caused by: ProtocolError(('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))))

I am using esrally : 2.10.0 and python : 3.9.18

2

Rally does not use TLS/SSL by default which might be causing that. Can you try adding use_ssl:true,ca_certs:'<path-to-pem-file>' to client options and comparing results? You can also try with verify-certs:false instead of ca_certs file path, as a quick test.

If this doesn't help, please share curl invocation that works.

3

Hi @gbanasiak ,

if i make verify-certs: false getting below error

esrally race --track=pmc --target-hosts=localhost:9200 --pipeline=benchmark-only --client-options="use_ssl:true,verify_certs:false,basic_auth_user:'elastic',basic_auth_password:'04CA3sd8FN7lYkfe6l27CW03'"


INFO] Race id is [1e93e917-6889-403e-bc15-2b5a94898093]
[ERROR] Cannot race. Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/esrally/actor.py", line 92, in guard
    return f(self, msg, sender)
  File "/usr/local/lib/python3.9/site-packages/esrally/racecontrol.py", line 111, in receiveMsg_Setup
    self.coordinator.setup(sources=msg.sources)
  File "/usr/local/lib/python3.9/site-packages/esrally/racecontrol.py", line 235, in setup
    self.metrics_store = metrics.metrics_store(
  File "/usr/local/lib/python3.9/site-packages/esrally/metrics.py", line 358, in metrics_store
    store = cls(cfg=cfg, meta_info=meta_info)
  File "/usr/local/lib/python3.9/site-packages/esrally/metrics.py", line 895, in __init__
    self._client = client_factory_class(cfg).create()
  File "/usr/local/lib/python3.9/site-packages/esrally/metrics.py", line 267, in __init__
    distribution_flavor, distribution_version, _, _ = client.cluster_distribution_version(
  File "/usr/local/lib/python3.9/site-packages/esrally/client/factory.py", line 349, in cluster_distribution_version
    version = es.info()["version"]
  File "/usr/local/lib/python3.9/site-packages/elasticsearch/_sync/client/utils.py", line 414, in wrapped
    return api(*args, **kwargs)
  File "/usr/local/lib/python3.9/site-packages/elasticsearch/_sync/client/__init__.py", line 2296, in info
    return self.perform_request(  # type: ignore[return-value]
  File "/usr/local/lib/python3.9/site-packages/esrally/client/synchronous.py", line 168, in perform_request
    info = self.transport.perform_request(method="GET", target="/", headers=request_headers)
  File "/usr/local/lib/python3.9/site-packages/elastic_transport/_transport.py", line 328, in perform_request
    meta, raw_data = node.perform_request(
  File "/usr/local/lib/python3.9/site-packages/elastic_transport/_node/_http_urllib3.py", line 197, in perform_request
    raise err from None
elastic_transport.TlsError: TLS error caused by: TlsError(TLS error caused by: SSLError([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate in certificate chain (_ssl.c:1129)))

if i pass the pem file , getting file not found error

esrally race --track=pmc --target-hosts=localhost:9200 --pipeline=benchmark-only --client-options="use_ssl:true,verify_certs:true, ca_certs:'/home/cloud-user/key.pem,basic_auth_user:'elastic',basic_auth_password:'04CA3sd8FN7lYkfe6l27CW03'"

[INFO] Race id is [ec2ef8eb-9230-49f5-b014-62a9b2728c63]
[ERROR] Cannot race. Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/esrally/actor.py", line 92, in guard
    return f(self, msg, sender)
  File "/usr/local/lib/python3.9/site-packages/esrally/racecontrol.py", line 111, in receiveMsg_Setup
    self.coordinator.setup(sources=msg.sources)
  File "/usr/local/lib/python3.9/site-packages/esrally/racecontrol.py", line 194, in setup
    ) = client.factory.cluster_distribution_version(hosts, client_options)
  File "/usr/local/lib/python3.9/site-packages/esrally/client/factory.py", line 345, in cluster_distribution_version
    es = client_factory(hosts, client_options).create()
  File "/usr/local/lib/python3.9/site-packages/esrally/client/factory.py", line 66, in __init__
    self.ssl_context = ssl.create_default_context(
  File "/usr/lib64/python3.9/ssl.py", line 746, in create_default_context
    context.load_verify_locations(cafile, capath, cadata)
FileNotFoundError: [Errno 2] No such file or directory

below is the curl response

[root@ipXX .rally]# curl -s -u "elastic:04CA3sd8FN7lYkfe6l27CW03" -k "https://localhost:9200/_cat/indices?h=index"
.geoip_databases
.security-7
test.02
test.01
4

With your first command you've run into TLS verification failure for your Elasticsearch metric store which should be a different cluster than the one you're testing.

Please try adding the following in [reporting] section in ~/.rally/rally.ini file (see doc):

datastore.ssl.verification_mode = none

Or disable Elasticsearch metric store completely for now.

Your second command has incorrect quotes for ca_certs client option I think.

5

hi @gbanasiak ,

yeah for second command i am getting like this now .

esrally race --track=pmc --target-hosts=localhost:9200 --pipeline=benchmark-only --client-options="use_ssl:true,verify_certs:true,ca_certs:'/home/cloud-user/.rally/key.pem'"

    ____        ____
   / __ \____ _/ / /_  __
  / /_/ / __ `/ / / / / /
 / _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
                /____/

[INFO] Race id is [a786ff9a-9b92-47e3-9162-b74ad9cf0200]
[ERROR] Cannot race. Traceback (most recent call last):
  File "/usr/local/lib/python3.9/site-packages/esrally/actor.py", line 92, in guard
    return f(self, msg, sender)
  File "/usr/local/lib/python3.9/site-packages/esrally/racecontrol.py", line 111, in receiveMsg_Setup
    self.coordinator.setup(sources=msg.sources)
  File "/usr/local/lib/python3.9/site-packages/esrally/racecontrol.py", line 194, in setup
    ) = client.factory.cluster_distribution_version(hosts, client_options)
  File "/usr/local/lib/python3.9/site-packages/esrally/client/factory.py", line 345, in cluster_distribution_version
    es = client_factory(hosts, client_options).create()
  File "/usr/local/lib/python3.9/site-packages/esrally/client/factory.py", line 66, in __init__
    self.ssl_context = ssl.create_default_context(
  File "/usr/lib64/python3.9/ssl.py", line 746, in create_default_context
    context.load_verify_locations(cafile, capath, cadata)
ssl.SSLError: [X509: NO_CERTIFICATE_OR_CRL_FOUND] no certificate or crl found (_ssl.c:4293)
6

The elasticsearch is running in kubernetes cluster . I did port forwarding on 9200 , where curl is working perfectly fine not the esrally .

bdw how to disable Elasticsearch metric store completely ? any steps we have ?

7

This suggests your /home/cloud-user/.rally/key.pem file has incorrect content. If you can make curl work without -k option providing certificates file with --cacert option, the same file should also work fine with Rally.

bdw how to disable Elasticsearch metric store completely ? any steps we have ?

Here is the default metric store configuration from rally.ini. It uses in-memory metric store as opposed to Elasticsearch metric store:

[reporting]
datastore.type = in-memory
datastore.host =
datastore.port =
datastore.secure = False
datastore.user =
datastore.password =
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.