SSL issue when running ESRally to ES hosted in a diff server

Elastic Stack Elasticsearch
1

i am having difficulty runny ESrally Race. i have import the .crt and update-ca-certificates to pem. but when i run the command, i am getting ssl error.

Commands tried:

esrally race --track=acme --target-hosts=x.x.x.x:9200 --client-options="use_ssl:true,verify_certs:true,basic_auth_user:'elastic',basic_auth_password:'changeme'"

esrally race --track=acme --target-hosts=x.x.x.x:9200 --client-options="use_ssl:true,verify_certs:false,basic_auth_user:'elastic',basic_auth_password:'changeme'"

esrally race --track=acme --target-hosts=x.x.x.x:9200 --client-options="use_ssl:true,verify_certs:true"

esrally race --track=acme --target-hosts=x.x.x.x:9200 --client-options="use_ssl:true,verify_certs:false"

esrally race --track=acme --target-hosts=x.x.x.x:9200 --client-options="basic_auth_user:'elastic',basic_auth_password:'changeme'"

Errors:

[INFO] Race id is [42e651c0-bfe1-46a4-b6d3-a31b4f8bbbd0]
[ERROR] Cannot race. Traceback (most recent call last):
  File "~/.local/lib/python3.10/site-packages/esrally/actor.py", line 92, in guard
    return f(self, msg, sender)
  File "~/.local/lib/python3.10/site-packages/esrally/racecontrol.py", line 114, in receiveMsg_Setup
    self.coordinator.setup(sources=msg.sources)
  File "~/.local/lib/python3.10/site-packages/esrally/racecontrol.py", line 203, in setup
    ) = client.factory.cluster_distribution_version(hosts, client_options)
  File "~/.local/lib/python3.10/site-packages/esrally/client/factory.py", line 359, in cluster_distribution_version
    version = es.info()["version"]
  File "~/.local/lib/python3.10/site-packages/elasticsearch/_sync/client/utils.py", line 414, in wrapped
    return api(*args, **kwargs)
  File "~/.local/lib/python3.10/site-packages/elasticsearch/_sync/client/__init__.py", line 2296, in info
    return self.perform_request(  # type: ignore[return-value]
  File "~/.local/lib/python3.10/site-packages/esrally/client/synchronous.py", line 168, in perform_request
    info = self.transport.perform_request(method="GET", target="/", headers=request_headers)
  File "~/.local/lib/python3.10/site-packages/elastic_transport/_transport.py", line 328, in perform_request
    meta, raw_data = node.perform_request(
  File "~/.local/lib/python3.10/site-packages/elastic_transport/_node/_http_urllib3.py", line 197, in perform_request
    raise err from None
elastic_transport.TlsError: TLS error caused by: TlsError(TLS error caused by: SSLError([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)))
2

Hello! Thank you for your interest in Rally.

Python ignores system certificates by default, using update-ca-certificates won't have any effect. (That said, there's recent work that allows system stores to work: truststore · PyPI. But that would be a request for the Rally developers.)

Are you getting the same error in each case? For example, use_ssl:true,verify_certs:false,basic_auth_user:'elastic',basic_auth_password:'changeme'" should not fail that way, as it won't try to verify the certificate.

To rule out Rally, can you please verify that connecting with curl works?

curl -k -u elastic:changeme https://x.x.x.x:9200/

You could also try Python:

import requests
r = requests.get("https://x.x.x.x:9200", verify=False, auth=("elastic", "changeme"))
print(r.json())

(Both approaches disable certificate verification, but we can make it work later if you want to.)

3

Hi @Quentin_Pradet ,

Yes, every command gave me the same error:

elastic_transport.TlsError: TLS error caused by: TlsError(TLS error caused by: SSLError([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)))

i tried curl -k -u elastic:changeme https://x.x.x.x:9200/ , it works. gave me the correct as the browser.

when i tried the python. i got the warning .local/lib/python3.10/site-packages/urllib3/connectionpool.py:1061: InsecureRequestWarning: Unverified HTTPS request is being made to host 'x.x.x.x' but still ran successfully. got the same response as above.

I am running ESRally locally on my laptop and my Elasticsearch is hosted in a remote network.

could it be my ESRally config?

[meta]
config.version = 17

[system]
env.name = dev

[rally]
es.host = https://x.x.x.x:9200/
es.username = elastic
es.password = changeme
es.api_key = apikey
es.ssl.verify = true
es.ssl.ca = /opt/app/elasticsearch-8.15.2/config/certs/http_ca.crt
track.data = /.rally/tracks
es.ssl.no-verify-certs = true

[ssl]
certfile = /etc/ssl/certs/es.pem

[node]
root.dir = /.rally/benchmarks
src.root.dir = /.rally/benchmarks/src

[source]
remote.repo.url = GitHub - elastic/elasticsearch: Free and Open Source, Distributed, RESTful Search Engine
elasticsearch.src.subdir = elasticsearch

[benchmarks]
local.dataset.cache = /.rally/benchmarks/data

[reporting]
datastore.type = elasticsearch
datastore.host = x.x.x.x
datastore.port = 9200
datastore.secure = True
datastore.user = elastic
datastore.password = changeme

[tracks]
default.url = GitHub - elastic/rally-tracks: Track specifications for the Elasticsearch benchmarking tool Rally

[teams]
default.url = GitHub - elastic/rally-teams: Default Elasticsearch configurations for the Elasticsearch benchmarking tool Rally

[defaults]
preserve_benchmark_candidate = false

[distributions]
release.cache = true

My ESRally is 2.11.0 and python 3.10. Thanks for the assistance

4

Hello @hiilmiee ,

Where did the following part of configuration came from? I think it's ignored, but I'm curious about the source.

[rally]
es.host = https://x.x.x.x:9200/
es.username = elastic
es.password = changeme
es.api_key = apikey
es.ssl.verify = true
es.ssl.ca = /opt/app/elasticsearch-8.15.2/config/certs/http_ca.crt
track.data = /.rally/tracks
es.ssl.no-verify-certs = true

[ssl]
certfile = /etc/ssl/certs/es.pem

Please try with --pipeline=benchmark-only option:

esrally race --track=acme --pipeline=benchmark-only --target-hosts=x.x.x.x:9200 --client-options="use_ssl:true,verify_certs:false,basic_auth_user:'elastic',basic_auth_password:'changeme'"

If this doesn't work, please share debug logs. To enable debug logs please edit ~/.rally/logging.json file as follows:

[..]
  "root": {
    "handlers": ["rally_log_handler"],
    "level": "DEBUG"
  },
[..]

Make sure there are no sensitive secrets in the file before sharing.

Thanks.

5

hi @gbanasiak ,

Those configs i got from chatgpt.

Please find below my logs:

2024-11-28 08:04:26,307 -not-actor-/PID:1363 esrally.rally INFO OS [uname_result(system='Linux', node='node1', release='5.15.146.1-microsoft-standard-WSL2', version='#1 SMP Thu Jan 11 04:09:03 UTC 2024', machine='x86_64')]
2024-11-28 08:04:26,307 -not-actor-/PID:1363 esrally.rally INFO Python [namespace(name='cpython', cache_tag='cpython-310', version=sys.version_info(major=3, minor=10, micro=12, releaselevel='final', serial=0), hexversion=50990320, _multiarch='x86_64-linux-gnu')]
2024-11-28 08:04:26,308 -not-actor-/PID:1363 esrally.rally INFO Rally version [2.11.0]
2024-11-28 08:04:26,308 -not-actor-/PID:1363 esrally.rally DEBUG Command line arguments: Namespace(subcommand='race', cluster_name='rally-benchmark', distribution_version='', team_path=None, team_repository='default', team_revision=None, race_id='a096d712-28df-4e26-abe5-ccaa591cd571', pipeline='benchmark-only', revision='current', track_repository='default', track_path=None, track_revision=None, track='acme', track_params='', challenge=None, car='defaults', car_params='', runtime_jdk=None, elasticsearch_plugins='', plugin_params='', target_hosts='https://x.x.x.x:9200', load_driver_hosts='localhost', client_options="use_ssl:true,verify_certs:false,basic_auth_user:'elastic',basic_auth_password:'changeme'", on_error='continue', telemetry='', telemetry_params='', distribution_repository='release', include_tasks=None, exclude_tasks=None, user_tag='', report_format='markdown', report_numbers_align='decimal', show_in_report='available', report_file='', preserve_install='false', test_mode=False, enable_driver_profiling=False, enable_assertions=False, kill_running_processes=False, source_build_method='default', effective_start_date=None, skip_rest_api_check=False, configuration_name=None, quiet=False, offline=False)
2024-11-28 08:04:26,308 -not-actor-/PID:1363 esrally.utils.net INFO Connecting directly to the Internet (no proxy support) for [all_proxy].
2024-11-28 08:04:26,308 -not-actor-/PID:1363 esrally.utils.net INFO Connecting directly to the Internet (no proxy support) for [all_proxy].
2024-11-28 08:04:26,308 -not-actor-/PID:1363 esrally.rally INFO Cleaning track dependency directory [~/.rally/libs]...
2024-11-28 08:04:26,315 -not-actor-/PID:1363 esrally.rally INFO Actor system already running locally? [False]
2024-11-28 08:04:26,315 -not-actor-/PID:1363 esrally.actor INFO Starting actor system with system base [multiprocTCPBase] and capabilities [{'coordinator': True, 'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900'}].
2024-11-28 08:04:26,355 -not-actor-/PID:1372 root INFO ++++ Actor System gen (3, 10) started, admin @ ActorAddr-(T|:1900)
2024-11-28 08:04:26,356 -not-actor-/PID:1372 root DEBUG Thespian source: ~/.local/lib/python3.10/site-packages/thespian/__init__.py
2024-11-28 08:04:26,371 -not-actor-/PID:1363 esrally.racecontrol INFO Race id is [a096d712-28df-4e26-abe5-ccaa591cd571]
2024-11-28 08:04:26,371 -not-actor-/PID:1363 esrally.racecontrol INFO User specified pipeline [benchmark-only].
2024-11-28 08:04:26,372 -not-actor-/PID:1363 esrally.racecontrol INFO Using configured hosts [{'host': 'x.x.x.x', 'port': 9200, 'use_ssl': True}]
2024-11-28 08:04:26,373 -not-actor-/PID:1363 esrally.actor DEBUG Creating new actor system with system base [multiprocTCPBase] on coordinator node.
2024-11-28 08:04:26,377 ActorAddr-(T|:1900)/PID:1372 esrally.actor DEBUG Capabilities [{'coordinator': True, 'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900', 'Thespian ActorSystem Name': 'multiprocTCPBase', 'Thespian ActorSystem Version': 2, 'Thespian Watch Supported': True, 'Python Version': (3, 10, 12, 'final', 0), 'Thespian Generation': (3, 10), 'Thespian Version': '1732781066325'}] match requirements [{'coordinator': True}].
2024-11-28 08:04:27,658 -not-actor-/PID:1374 elastic_transport.node_pool WARNING Node <Urllib3HttpNode(https://x.x.x.x:9200)> has failed for 1 times in a row, putting on 1 second timeout
2024-11-28 08:04:27,658 -not-actor-/PID:1374 elastic_transport.transport WARNING Retrying request after failure (attempt 0 of 3)
Traceback (most recent call last):
  File "~/.local/lib/python3.10/site-packages/elastic_transport/_transport.py", line 328, in perform_request
    meta, raw_data = node.perform_request(
  File "~/.local/lib/python3.10/site-packages/elastic_transport/_node/_http_urllib3.py", line 197, in perform_request
    raise err from None
elastic_transport.TlsError: TLS error caused by: SSLError([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007))
2024-11-28 08:04:27,753 -not-actor-/PID:1374 elastic_transport.node_pool WARNING Node <Urllib3HttpNode(https://x.x.x.x:9200)> has failed for 2 times in a row, putting on 2 second timeout
2024-11-28 08:04:27,753 -not-actor-/PID:1374 elastic_transport.transport WARNING Retrying request after failure (attempt 1 of 3)
Traceback (most recent call last):
  File "~/.local/lib/python3.10/site-packages/elastic_transport/_transport.py", line 328, in perform_request
    meta, raw_data = node.perform_request(
  File "~/.local/lib/python3.10/site-packages/elastic_transport/_node/_http_urllib3.py", line 197, in perform_request
    raise err from None
elastic_transport.TlsError: TLS error caused by: SSLError([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007))
2024-11-28 08:04:27,842 -not-actor-/PID:1374 elastic_transport.node_pool WARNING Node <Urllib3HttpNode(https://x.x.x.x:9200)> has failed for 3 times in a row, putting on 4 second timeout
2024-11-28 08:04:27,842 -not-actor-/PID:1374 elastic_transport.transport WARNING Retrying request after failure (attempt 2 of 3)
Traceback (most recent call last):
  File "~/.local/lib/python3.10/site-packages/elastic_transport/_transport.py", line 328, in perform_request
    meta, raw_data = node.perform_request(
  File "~/.local/lib/python3.10/site-packages/elastic_transport/_node/_http_urllib3.py", line 197, in perform_request
    raise err from None
elastic_transport.TlsError: TLS error caused by: SSLError([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007))
2024-11-28 08:04:27,934 -not-actor-/PID:1374 elastic_transport.node_pool WARNING Node <Urllib3HttpNode(https://x.x.x.x:9200)> has failed for 4 times in a row, putting on 8 second timeout
2024-11-28 08:04:26,397 ActorAddr-(T|:60515)/PID:1374 esrally.client.factory INFO Creating ES client connected to [{'host': 'x.x.x.x', 'port': 9200, 'use_ssl': True}] with options [{'timeout': 60, 'use_ssl': True, 'verify_certs': False, 'basic_auth_user': 'elastic', 'basic_auth_password': '*****'}]
2024-11-28 08:04:26,397 ActorAddr-(T|:60515)/PID:1374 esrally.client.factory DEBUG SSL support: on
2024-11-28 08:04:26,470 ActorAddr-(T|:60515)/PID:1374 esrally.client.factory DEBUG SSL certificate verification: off
2024-11-28 08:04:26,470 ActorAddr-(T|:60515)/PID:1374 esrally.client.factory WARNING User has enabled SSL but disabled certificate verification. This is dangerous but may be ok for a benchmark.
2024-11-28 08:04:26,470 ActorAddr-(T|:60515)/PID:1374 esrally.client.factory DEBUG SSL client authentication: off
2024-11-28 08:04:26,470 ActorAddr-(T|:60515)/PID:1374 esrally.client.factory DEBUG Automatic creation of client API keys: off
2024-11-28 08:04:26,471 ActorAddr-(T|:60515)/PID:1374 esrally.client.factory DEBUG HTTP basic authentication: on
2024-11-28 08:04:26,471 ActorAddr-(T|:60515)/PID:1374 esrally.client.factory DEBUG API key authentication: off
2024-11-28 08:04:26,471 ActorAddr-(T|:60515)/PID:1374 esrally.client.factory DEBUG HTTP compression: off
2024-11-28 08:04:26,608 ActorAddr-(T|:60515)/PID:1374 esrally.client.factory DEBUG REST API is available for >= [1] nodes after [1] attempts.
2024-11-28 08:04:26,608 ActorAddr-(T|:60515)/PID:1374 esrally.racecontrol INFO Automatically derived distribution flavor [default], version [8.15.2], and build hash [98adf7bf6bb69b66ab95b761c9e5aadb0bb059a3]
2024-11-28 08:04:26,609 ActorAddr-(T|:60515)/PID:1374 esrally.utils.process DEBUG Running subprocess [git -C ~/.rally/benchmarks/tracks/default --version] with logging.
2024-11-28 08:04:26,612 ActorAddr-(T|:60515)/PID:1374 esrally.utils.process DEBUG git version 2.34.1

2024-11-28 08:04:26,614 ActorAddr-(T|:60515)/PID:1374 esrally.utils.process DEBUG Subprocess [git -C ~/.rally/benchmarks/tracks/default --version] finished with return code [0].
2024-11-28 08:04:26,614 ActorAddr-(T|:60515)/PID:1374 esrally.utils.process DEBUG Running subprocess [git -C ~/.rally/benchmarks/tracks/default fetch --prune --tags origin] with logging.
2024-11-28 08:04:27,217 ActorAddr-(T|:60515)/PID:1374 esrally.utils.process DEBUG Subprocess [git -C ~/.rally/benchmarks/tracks/default fetch --prune --tags origin] finished with return code [0].
2024-11-28 08:04:27,217 ActorAddr-(T|:60515)/PID:1374 esrally.utils.process DEBUG Running subprocess [git -C ~/.rally/benchmarks/tracks/default --version] with logging.
2024-11-28 08:04:27,219 ActorAddr-(T|:60515)/PID:1374 esrally.utils.process DEBUG git version 2.34.1

2024-11-28 08:04:27,219 ActorAddr-(T|:60515)/PID:1374 esrally.utils.process DEBUG Subprocess [git -C ~/.rally/benchmarks/tracks/default --version] finished with return code [0].
2024-11-28 08:04:27,219 ActorAddr-(T|:60515)/PID:1374 esrally.utils.process DEBUG Running subprocess [git -C ~/.rally/benchmarks/tracks/default for-each-ref refs/remotes/ --format='%(refname:short)'] with output.
2024-11-28 08:04:27,223 ActorAddr-(T|:60515)/PID:1374 esrally.utils.repo INFO Checking out [8.15] in [~/.rally/benchmarks/tracks/default] for distribution version [8.15.2].
2024-11-28 08:04:27,223 ActorAddr-(T|:60515)/PID:1374 esrally.utils.process DEBUG Running subprocess [git -C ~/.rally/benchmarks/tracks/default --version] with logging.
2024-11-28 08:04:27,225 ActorAddr-(T|:60515)/PID:1374 esrally.utils.process DEBUG git version 2.34.1

2024-11-28 08:04:27,226 ActorAddr-(T|:60515)/PID:1374 esrally.utils.process DEBUG Subprocess [git -C ~/.rally/benchmarks/tracks/default --version] finished with return code [0].
2024-11-28 08:04:27,226 ActorAddr-(T|:60515)/PID:1374 esrally.utils.process DEBUG Running subprocess [git -C ~/.rally/benchmarks/tracks/default checkout 8.15] with logging.
2024-11-28 08:04:27,403 ActorAddr-(T|:60515)/PID:1374 esrally.utils.process INFO Switched to branch '8.15'
Your branch is up to date with 'origin/8.15'.

2024-11-28 08:04:27,403 ActorAddr-(T|:60515)/PID:1374 esrally.utils.process DEBUG Subprocess [git -C ~/.rally/benchmarks/tracks/default checkout 8.15] finished with return code [0].
2024-11-28 08:04:27,403 ActorAddr-(T|:60515)/PID:1374 esrally.utils.repo INFO Rebasing on [8.15] in [~/.rally/benchmarks/tracks/default] for distribution version [8.15.2].
2024-11-28 08:04:27,403 ActorAddr-(T|:60515)/PID:1374 esrally.utils.process DEBUG Running subprocess [git -C ~/.rally/benchmarks/tracks/default --version] with logging.
2024-11-28 08:04:27,406 ActorAddr-(T|:60515)/PID:1374 esrally.utils.process DEBUG git version 2.34.1

2024-11-28 08:04:27,407 ActorAddr-(T|:60515)/PID:1374 esrally.utils.process DEBUG Subprocess [git -C ~/.rally/benchmarks/tracks/default --version] finished with return code [0].
2024-11-28 08:04:27,407 ActorAddr-(T|:60515)/PID:1374 esrally.utils.process DEBUG Running subprocess [git -C ~/.rally/benchmarks/tracks/default --version] with logging.
2024-11-28 08:04:27,410 ActorAddr-(T|:60515)/PID:1374 esrally.utils.process DEBUG git version 2.34.1

2024-11-28 08:04:27,410 ActorAddr-(T|:60515)/PID:1374 esrally.utils.process DEBUG Subprocess [git -C ~/.rally/benchmarks/tracks/default --version] finished with return code [0].
2024-11-28 08:04:27,410 ActorAddr-(T|:60515)/PID:1374 esrally.utils.process DEBUG Running subprocess [git -C ~/.rally/benchmarks/tracks/default checkout 8.15] with logging.
2024-11-28 08:04:27,422 ActorAddr-(T|:60515)/PID:1374 esrally.utils.process INFO Already on '8.15'
Your branch is up to date with 'origin/8.15'.

2024-11-28 08:04:27,422 ActorAddr-(T|:60515)/PID:1374 esrally.utils.process DEBUG Subprocess [git -C ~/.rally/benchmarks/tracks/default checkout 8.15] finished with return code [0].
2024-11-28 08:04:27,422 ActorAddr-(T|:60515)/PID:1374 esrally.utils.process DEBUG Running subprocess [git -C ~/.rally/benchmarks/tracks/default rebase origin/8.15] with logging.
2024-11-28 08:04:27,434 ActorAddr-(T|:60515)/PID:1374 esrally.utils.process INFO Current branch 8.15 is up to date.

2024-11-28 08:04:27,434 ActorAddr-(T|:60515)/PID:1374 esrally.utils.process DEBUG Subprocess [git -C ~/.rally/benchmarks/tracks/default rebase origin/8.15] finished with return code [0].
2024-11-28 08:04:27,434 ActorAddr-(T|:60515)/PID:1374 esrally.utils.process DEBUG Running subprocess [git -C ~/.rally/benchmarks/tracks/default --version] with logging.
2024-11-28 08:04:27,436 ActorAddr-(T|:60515)/PID:1374 esrally.utils.process DEBUG git version 2.34.1

2024-11-28 08:04:27,436 ActorAddr-(T|:60515)/PID:1374 esrally.utils.process DEBUG Subprocess [git -C ~/.rally/benchmarks/tracks/default --version] finished with return code [0].
2024-11-28 08:04:27,436 ActorAddr-(T|:60515)/PID:1374 esrally.utils.process DEBUG Running subprocess [git -C ~/.rally/benchmarks/tracks/default rev-parse HEAD] with output.
2024-11-28 08:04:27,996 -not-actor-/PID:1363 esrally.racecontrol ERROR A benchmark failure has occurred
2024-11-28 08:04:27,997 -not-actor-/PID:1363 esrally.racecontrol INFO Telling benchmark actor to exit.
2024-11-28 08:04:27,439 ActorAddr-(T|:60515)/PID:1374 esrally.track.loader INFO Reading track specification file [~/.rally/benchmarks/tracks/default/acme/track.json].
2024-11-28 08:04:27,457 ActorAddr-(T|:60515)/PID:1374 esrally.track.loader INFO Final rendered track for '~/.rally/benchmarks/tracks/default/acme/track.json' has been written to '/tmp/tmp0lhfphw7.json'.
2024-11-28 08:04:27,468 ActorAddr-(T|:60515)/PID:1374 esrally.track.loader INFO Loading template [definition for index search-ocr-poc in search-ocr-poc.json].
2024-11-28 08:04:27,481 ActorAddr-(T|:60515)/PID:1374 esrally.track.loader DEBUG Using built-in operation type [search] for operation [default].
2024-11-28 08:04:27,482 ActorAddr-(T|:60515)/PID:1374 esrally.track.loader DEBUG Using built-in operation type [delete-index] for operation [delete-index].
2024-11-28 08:04:27,482 ActorAddr-(T|:60515)/PID:1374 esrally.track.loader DEBUG Using built-in operation type [create-index] for operation [create-index].
2024-11-28 08:04:27,483 ActorAddr-(T|:60515)/PID:1374 esrally.track.loader DEBUG Using built-in operation type [cluster-health] for operation [cluster-health].
2024-11-28 08:04:27,483 ActorAddr-(T|:60515)/PID:1374 esrally.track.loader DEBUG Using built-in operation type [bulk] for operation [bulk].
2024-11-28 08:04:27,484 ActorAddr-(T|:60515)/PID:1374 esrally.track.loader DEBUG Invoking plugin_reader with name [acme] resolved to path [~/.rally/benchmarks/tracks/default/acme]
2024-11-28 08:04:27,484 ActorAddr-(T|:60515)/PID:1374 esrally.client.factory INFO Creating ES client connected to [{'host': 'x.x.x.x', 'port': '9200'}] with options [{'use_ssl': True, 'verify_certs': True, 'timeout': 120, 'basic_auth_user': 'elastic', 'basic_auth_password': '*****'}]
2024-11-28 08:04:27,485 ActorAddr-(T|:60515)/PID:1374 esrally.client.factory DEBUG SSL support: on
2024-11-28 08:04:27,565 ActorAddr-(T|:60515)/PID:1374 esrally.client.factory DEBUG SSL certificate verification: on
2024-11-28 08:04:27,566 ActorAddr-(T|:60515)/PID:1374 esrally.client.factory DEBUG SSL client authentication: off
2024-11-28 08:04:27,566 ActorAddr-(T|:60515)/PID:1374 esrally.client.factory DEBUG Automatic creation of client API keys: off
2024-11-28 08:04:27,566 ActorAddr-(T|:60515)/PID:1374 esrally.client.factory DEBUG HTTP basic authentication: on
2024-11-28 08:04:27,566 ActorAddr-(T|:60515)/PID:1374 esrally.client.factory DEBUG API key authentication: off
2024-11-28 08:04:27,566 ActorAddr-(T|:60515)/PID:1374 esrally.client.factory DEBUG HTTP compression: off
2024-11-28 08:04:27,935 ActorAddr-(T|:60515)/PID:1374 esrally.actor ERROR Error in race control
Traceback (most recent call last):

  File "~/.local/lib/python3.10/site-packages/esrally/actor.py", line 92, in guard
    return f(self, msg, sender)

  File "~/.local/lib/python3.10/site-packages/esrally/racecontrol.py", line 114, in receiveMsg_Setup
    self.coordinator.setup(sources=msg.sources)

  File "~/.local/lib/python3.10/site-packages/esrally/racecontrol.py", line 244, in setup
    self.metrics_store = metrics.metrics_store(

  File "~/.local/lib/python3.10/site-packages/esrally/metrics.py", line 354, in metrics_store
    store = cls(cfg=cfg, meta_info=meta_info)

  File "~/.local/lib/python3.10/site-packages/esrally/metrics.py", line 891, in __init__
    self._client = client_factory_class(cfg).create()

  File "~/.local/lib/python3.10/site-packages/esrally/metrics.py", line 263, in __init__
    distribution_flavor, distribution_version, _, _ = client.cluster_distribution_version(

  File "~/.local/lib/python3.10/site-packages/esrally/client/factory.py", line 359, in cluster_distribution_version
    version = es.info()["version"]

  File "~/.local/lib/python3.10/site-packages/elasticsearch/_sync/client/utils.py", line 414, in wrapped
    return api(*args, **kwargs)

  File "~/.local/lib/python3.10/site-packages/elasticsearch/_sync/client/__init__.py", line 2296, in info
    return self.perform_request(  # type: ignore[return-value]

  File "~/.local/lib/python3.10/site-packages/esrally/client/synchronous.py", line 168, in perform_request
    info = self.transport.perform_request(method="GET", target="/", headers=request_headers)

  File "~/.local/lib/python3.10/site-packages/elastic_transport/_transport.py", line 328, in perform_request
    meta, raw_data = node.perform_request(

  File "~/.local/lib/python3.10/site-packages/elastic_transport/_node/_http_urllib3.py", line 197, in perform_request
    raise err from None

elastic_transport.TlsError: TLS error caused by: TlsError(TLS error caused by: SSLError([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)))

2024-11-28 08:04:27,998 ActorAddr-(T|:60515)/PID:1374 esrally.actor DEBUG BenchmarkActor received unknown message [ActorExitRequest] (ignoring).
2024-11-28 08:04:31,2 -not-actor-/PID:1363 esrally.rally INFO Attempting to shutdown internal actor system.
2024-11-28 08:04:31,4 -not-actor-/PID:1373 root INFO ActorSystem Logging Shutdown
2024-11-28 08:04:31,25 -not-actor-/PID:1372 root INFO ---- Actor System shutdown
2024-11-28 08:04:31,26 -not-actor-/PID:1363 esrally.rally INFO Shutdown completed.
2024-11-28 08:04:31,26 -not-actor-/PID:1363 esrally.rally ERROR Cannot run subcommand [race].
Traceback (most recent call last):
  File "~/.local/lib/python3.10/site-packages/esrally/rally.py", line 1189, in dispatch_sub_command
    race(cfg, args.kill_running_processes)
  File "~/.local/lib/python3.10/site-packages/esrally/rally.py", line 937, in race
    with_actor_system(racecontrol.run, cfg)
  File "~/.local/lib/python3.10/site-packages/esrally/rally.py", line 967, in with_actor_system
    runnable(cfg)
  File "~/.local/lib/python3.10/site-packages/esrally/racecontrol.py", line 408, in run
    raise e
  File "~/.local/lib/python3.10/site-packages/esrally/racecontrol.py", line 405, in run
    pipeline(cfg)
  File "~/.local/lib/python3.10/site-packages/esrally/racecontrol.py", line 74, in __call__
    self.target(cfg)
  File "~/.local/lib/python3.10/site-packages/esrally/racecontrol.py", line 344, in benchmark_only
    return race(cfg, external=True)
  File "~/.local/lib/python3.10/site-packages/esrally/racecontrol.py", line 302, in race
    raise exceptions.RallyError(result.message, result.cause)
esrally.exceptions.RallyError: Traceback (most recent call last):
  File "~/.local/lib/python3.10/site-packages/esrally/actor.py", line 92, in guard
    return f(self, msg, sender)
  File "~/.local/lib/python3.10/site-packages/esrally/racecontrol.py", line 114, in receiveMsg_Setup
    self.coordinator.setup(sources=msg.sources)
  File "~/.local/lib/python3.10/site-packages/esrally/racecontrol.py", line 244, in setup
    self.metrics_store = metrics.metrics_store(
  File "~/.local/lib/python3.10/site-packages/esrally/metrics.py", line 354, in metrics_store
    store = cls(cfg=cfg, meta_info=meta_info)
  File "~/.local/lib/python3.10/site-packages/esrally/metrics.py", line 891, in __init__
    self._client = client_factory_class(cfg).create()
  File "~/.local/lib/python3.10/site-packages/esrally/metrics.py", line 263, in __init__
    distribution_flavor, distribution_version, _, _ = client.cluster_distribution_version(
  File "~/.local/lib/python3.10/site-packages/esrally/client/factory.py", line 359, in cluster_distribution_version
    version = es.info()["version"]
  File "~/.local/lib/python3.10/site-packages/elasticsearch/_sync/client/utils.py", line 414, in wrapped
    return api(*args, **kwargs)
  File "~/.local/lib/python3.10/site-packages/elasticsearch/_sync/client/__init__.py", line 2296, in info
    return self.perform_request(  # type: ignore[return-value]
  File "~/.local/lib/python3.10/site-packages/esrally/client/synchronous.py", line 168, in perform_request
    info = self.transport.perform_request(method="GET", target="/", headers=request_headers)
  File "~/.local/lib/python3.10/site-packages/elastic_transport/_transport.py", line 328, in perform_request
    meta, raw_data = node.perform_request(
  File "~/.local/lib/python3.10/site-packages/elastic_transport/_node/_http_urllib3.py", line 197, in perform_request
    raise err from None
elastic_transport.TlsError: TLS error caused by: TlsError(TLS error caused by: SSLError([SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1007)))
6

Based on the logs, the TLS error comes from your connection to the metric store as per your [reporting] configuration section. There is a separate Elasticsearch client created for this, which still verifies TLS cert.

Please add datastore.ssl.verification_mode = none in [reporting].

We recommend sending metrics to a different Elasticsearch cluster than the tested one.

7

hi @gbanasiak ,

seems like it works fine now. thanks ! also, what did u mean by different in this statement:

We recommend sending metrics to a different Elasticsearch cluster than the tested one.

also, if i run the command without --pipeline=benchmark-only, i seem to get an error :

esrally.exceptions.SystemSetupError: When specifying nodes to be managed by Rally you can only supply hostname:port pairs (e.g. 'localhost:9200'), any additional options cannot be supported.

and the log shows:

2024-12-02 07:08:12,890 -not-actor-/PID:2016 esrally.rally INFO OS [uname_result(system='Linux', node='node1', release='5.15.146.1-microsoft-standard-WSL2', version='#1 SMP Thu Jan 11 04:09:03 UTC 2024', machine='x86_64')]
2024-12-02 07:08:12,890 -not-actor-/PID:2016 1.rally INFO Python [namespace(name='cpython', cache_tag='cpython-310', version=sys.version_info(major=3, minor=10, micro=12, releaselevel='final', serial=0), hexversion=50990320, _multiarch='x86_64-linux-gnu')]
2024-12-02 07:08:12,890 -not-actor-/PID:2016 esrally.rally INFO Rally version [2.11.0]
2024-12-02 07:08:12,890 -not-actor-/PID:2016 esrally.rally DEBUG Command line arguments: Namespace(subcommand='race', cluster_name='rally-benchmark', distribution_version='', team_path=None, team_repository='default', team_revision=None, race_id='ac3ff5ba-428f-4f32-8397-5a2e2c68a790', pipeline='', revision='current', track_repository='default', track_path='~/.rally/benchmarks/tracks/default/acme/', track_revision=None, track=None, track_params='', challenge=None, car='defaults', car_params='', runtime_jdk=None, elasticsearch_plugins='', plugin_params='', target_hosts='https://x.x.x.x:9200', load_driver_hosts='localhost', client_options="use_ssl:true,verify_certs:false,basic_auth_user:'elastic',basic_auth_password:'*****'", on_error='continue', telemetry='', telemetry_params='', distribution_repository='release', include_tasks=None, exclude_tasks=None, user_tag='', report_format='markdown', report_numbers_align='decimal', show_in_report='available', report_file='', preserve_install='false', test_mode=False, enable_driver_profiling=False, enable_assertions=False, kill_running_processes=False, source_build_method='default', effective_start_date=None, skip_rest_api_check=False, configuration_name=None, quiet=False, offline=False)
2024-12-02 07:08:12,890 -not-actor-/PID:2016 esrally.utils.net INFO Connecting directly to the Internet (no proxy support) for [all_proxy].
2024-12-02 07:08:12,890 -not-actor-/PID:2016 esrally.utils.net INFO Connecting directly to the Internet (no proxy support) for [all_proxy].
2024-12-02 07:08:12,890 -not-actor-/PID:2016 esrally.rally INFO Cleaning track dependency directory [/~/.rally/libs]...
2024-12-02 07:08:12,895 -not-actor-/PID:2016 esrally.rally INFO Actor system already running locally? [False]
2024-12-02 07:08:12,895 -not-actor-/PID:2016 esrally.actor INFO Starting actor system with system base [multiprocTCPBase] and capabilities [{'coordinator': True, 'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900'}].
2024-12-02 07:08:12,921 -not-actor-/PID:2025 root INFO ++++ Actor System gen (3, 10) started, admin @ ActorAddr-(T|:1900)
2024-12-02 07:08:12,921 -not-actor-/PID:2025 root DEBUG Thespian source: /~/.local/lib/python3.10/site-packages/thespian/__init__.py
2024-12-02 07:08:12,937 -not-actor-/PID:2016 esrally.racecontrol INFO Race id is [ac3ff5ba-428f-4f32-8397-5a2e2c68a790]
2024-12-02 07:08:12,938 -not-actor-/PID:2016 esrally.racecontrol INFO User specified no pipeline. Automatically derived pipeline [from-sources].
2024-12-02 07:08:12,938 -not-actor-/PID:2016 esrally.racecontrol INFO Using configured hosts [{'host': 'x.x.x.x', 'port': 9200, 'use_ssl': True}]
2024-12-02 07:08:12,939 -not-actor-/PID:2016 esrally.actor DEBUG Creating new actor system with system base [multiprocTCPBase] on coordinator node.
2024-12-02 07:08:12,944 ActorAddr-(T|:1900)/PID:2025 esrally.actor DEBUG Capabilities [{'coordinator': True, 'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900', 'Thespian ActorSystem Name': 'multiprocTCPBase', 'Thespian ActorSystem Version': 2, 'Thespian Watch Supported': True, 'Python Version': (3, 10, 12, 'final', 0), 'Thespian Generation': (3, 10), 'Thespian Version': '1733123292900'}] match requirements [{'coordinator': True}].
2024-12-02 07:08:12,962 ActorAddr-(T|:60821)/PID:2027 esrally.track.loader INFO Reading track specification file [/~/.rally/benchmarks/tracks/default/acme/track.json].
2024-12-02 07:08:12,977 ActorAddr-(T|:60821)/PID:2027 esrally.track.loader INFO Final rendered track for '/~/.rally/benchmarks/tracks/default/acme/track.json' has been written to '/tmp/tmpyinu2l7s.json'.
2024-12-02 07:08:12,982 ActorAddr-(T|:60821)/PID:2027 esrally.track.loader INFO Loading template [definition for index search-ocr-poc in search-ocr-poc.json].
2024-12-02 07:08:12,988 ActorAddr-(T|:60821)/PID:2027 esrally.track.loader DEBUG Using built-in operation type [search] for operation [default].
2024-12-02 07:08:12,989 ActorAddr-(T|:60821)/PID:2027 esrally.track.loader DEBUG Using built-in operation type [delete-index] for operation [delete-index].
2024-12-02 07:08:12,989 ActorAddr-(T|:60821)/PID:2027 esrally.track.loader DEBUG Using built-in operation type [create-index] for operation [create-index].
2024-12-02 07:08:12,989 ActorAddr-(T|:60821)/PID:2027 esrally.track.loader DEBUG Using built-in operation type [cluster-health] for operation [cluster-health].
2024-12-02 07:08:12,989 ActorAddr-(T|:60821)/PID:2027 esrally.track.loader DEBUG Using built-in operation type [bulk] for operation [bulk].
2024-12-02 07:08:12,989 ActorAddr-(T|:60821)/PID:2027 esrally.track.loader DEBUG Invoking plugin_reader with name [acme] resolved to path [/~/.rally/benchmarks/tracks/default/acme]
2024-12-02 07:08:12,991 ActorAddr-(T|:60821)/PID:2027 esrally.client.factory INFO Creating ES client connected to [{'host': 'x.x.x.x', 'port': '9200'}] with options [{'use_ssl': True, 'verify_certs': False, 'timeout': 120, 'basic_auth_user': 'elastic', 'basic_auth_password': '*****'}]
2024-12-02 07:08:12,991 ActorAddr-(T|:60821)/PID:2027 esrally.client.factory DEBUG SSL support: on
2024-12-02 07:08:13,31 ActorAddr-(T|:60821)/PID:2027 esrally.client.factory DEBUG SSL certificate verification: off
2024-12-02 07:08:13,31 ActorAddr-(T|:60821)/PID:2027 esrally.client.factory WARNING User has enabled SSL but disabled certificate verification. This is dangerous but may be ok for a benchmark.
2024-12-02 07:08:13,31 ActorAddr-(T|:60821)/PID:2027 esrally.client.factory DEBUG SSL client authentication: off
2024-12-02 07:08:13,31 ActorAddr-(T|:60821)/PID:2027 esrally.client.factory DEBUG Automatic creation of client API keys: off
2024-12-02 07:08:13,32 ActorAddr-(T|:60821)/PID:2027 esrally.client.factory DEBUG HTTP basic authentication: on
2024-12-02 07:08:13,32 ActorAddr-(T|:60821)/PID:2027 esrally.client.factory DEBUG API key authentication: off
2024-12-02 07:08:13,32 ActorAddr-(T|:60821)/PID:2027 esrally.client.factory DEBUG HTTP compression: off
2024-12-02 07:08:13,218 ActorAddr-(T|:60821)/PID:2027 esrally.client.factory DEBUG REST API is available for >= [1] nodes after [1] attempts.
2024-12-02 07:08:13,218 ActorAddr-(T|:60821)/PID:2027 esrally.client.factory INFO Creating ES client connected to [{'host': 'x.x.x.x', 'port': '9200'}] with options [{'use_ssl': True, 'verify_certs': False, 'timeout': 120, 'basic_auth_user': 'elastic', 'basic_auth_password': '*****'}]
2024-12-02 07:08:13,218 ActorAddr-(T|:60821)/PID:2027 esrally.client.factory DEBUG SSL support: on
2024-12-02 07:08:13,258 ActorAddr-(T|:60821)/PID:2027 esrally.client.factory DEBUG SSL certificate verification: off
2024-12-02 07:08:13,258 ActorAddr-(T|:60821)/PID:2027 esrally.client.factory WARNING User has enabled SSL but disabled certificate verification. This is dangerous but may be ok for a benchmark.
2024-12-02 07:08:13,259 ActorAddr-(T|:60821)/PID:2027 esrally.client.factory DEBUG SSL client authentication: off
2024-12-02 07:08:13,259 ActorAddr-(T|:60821)/PID:2027 esrally.client.factory DEBUG Automatic creation of client API keys: off
2024-12-02 07:08:13,259 ActorAddr-(T|:60821)/PID:2027 esrally.client.factory DEBUG HTTP basic authentication: on
2024-12-02 07:08:13,259 ActorAddr-(T|:60821)/PID:2027 esrally.client.factory DEBUG API key authentication: off
2024-12-02 07:08:13,259 ActorAddr-(T|:60821)/PID:2027 esrally.client.factory DEBUG HTTP compression: off
2024-12-02 07:08:13,260 ActorAddr-(T|:60821)/PID:2027 esrally.metrics INFO Creating Elasticsearch metrics store
2024-12-02 07:08:13,261 ActorAddr-(T|:60821)/PID:2027 esrally.metrics INFO Opening metrics store for race timestamp=[20241202T070812Z], track=[acme], challenge=[my-challenge], car=[['defaults']]
2024-12-02 07:08:13,311 ActorAddr-(T|:60821)/PID:2027 esrally.metrics INFO [rally-metrics-2024-12] already exists.
2024-12-02 07:08:13,317 ActorAddr-(T|:60821)/PID:2027 esrally.metrics INFO Creating ES race store
2024-12-02 07:08:13,318 ActorAddr-(T|:60821)/PID:2027 esrally.client.factory INFO Creating ES client connected to [{'host': 'x.x.x.x', 'port': '9200'}] with options [{'use_ssl': True, 'verify_certs': False, 'timeout': 120, 'basic_auth_user': 'elastic', 'basic_auth_password': '*****'}]
2024-12-02 07:08:13,318 ActorAddr-(T|:60821)/PID:2027 esrally.client.factory DEBUG SSL support: on
2024-12-02 07:08:13,360 ActorAddr-(T|:60821)/PID:2027 esrally.client.factory DEBUG SSL certificate verification: off
2024-12-02 07:08:13,361 ActorAddr-(T|:60821)/PID:2027 esrally.client.factory WARNING User has enabled SSL but disabled certificate verification. This is dangerous but may be ok for a benchmark.
2024-12-02 07:08:13,361 ActorAddr-(T|:60821)/PID:2027 esrally.client.factory DEBUG SSL client authentication: off
2024-12-02 07:08:13,361 ActorAddr-(T|:60821)/PID:2027 esrally.client.factory DEBUG Automatic creation of client API keys: off
2024-12-02 07:08:13,361 ActorAddr-(T|:60821)/PID:2027 esrally.client.factory DEBUG HTTP basic authentication: on
2024-12-02 07:08:13,361 ActorAddr-(T|:60821)/PID:2027 esrally.client.factory DEBUG API key authentication: off
2024-12-02 07:08:13,361 ActorAddr-(T|:60821)/PID:2027 esrally.client.factory DEBUG HTTP compression: off
2024-12-02 07:08:13,399 ActorAddr-(T|:60821)/PID:2027 esrally.client.factory DEBUG REST API is available for >= [1] nodes after [1] attempts.
2024-12-02 07:08:13,399 ActorAddr-(T|:60821)/PID:2027 esrally.client.factory INFO Creating ES client connected to [{'host': 'x.x.x.x', 'port': '9200'}] with options [{'use_ssl': True, 'verify_certs': False, 'timeout': 120, 'basic_auth_user': 'elastic', 'basic_auth_password': '*****'}]
2024-12-02 07:08:13,399 ActorAddr-(T|:60821)/PID:2027 esrally.client.factory DEBUG SSL support: on
2024-12-02 07:08:13,443 ActorAddr-(T|:60821)/PID:2027 esrally.client.factory DEBUG SSL certificate verification: off
2024-12-02 07:08:13,444 ActorAddr-(T|:60821)/PID:2027 esrally.client.factory WARNING User has enabled SSL but disabled certificate verification. This is dangerous but may be ok for a benchmark.
2024-12-02 07:08:13,444 ActorAddr-(T|:60821)/PID:2027 esrally.client.factory DEBUG SSL client authentication: off
2024-12-02 07:08:13,444 ActorAddr-(T|:60821)/PID:2027 esrally.client.factory DEBUG Automatic creation of client API keys: off
2024-12-02 07:08:13,444 ActorAddr-(T|:60821)/PID:2027 esrally.client.factory DEBUG HTTP basic authentication: on
2024-12-02 07:08:13,444 ActorAddr-(T|:60821)/PID:2027 esrally.client.factory DEBUG API key authentication: off
2024-12-02 07:08:13,444 ActorAddr-(T|:60821)/PID:2027 esrally.client.factory DEBUG HTTP compression: off
2024-12-02 07:08:13,445 ActorAddr-(T|:60821)/PID:2027 esrally.actor INFO Asking mechanic to start the engine.
2024-12-02 07:08:13,445 ActorAddr-(T|:60821)/PID:2027 esrally.actor DEBUG Capabilities [{'coordinator': True, 'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900', 'Thespian ActorSystem Name': 'multiprocTCPBase', 'Thespian ActorSystem Version': 2, 'Thespian Watch Supported': True, 'Python Version': (3, 10, 12, 'final', 0), 'Thespian Generation': (3, 10), 'Thespian Version': '1733123292900'}] match requirements [{'coordinator': True}].
2024-12-02 07:08:13,481 ActorAddr-(T|:60793)/PID:2028 esrally.actor INFO Received signal from race control to start engine.
2024-12-02 07:08:13,482 ActorAddr-(T|:60793)/PID:2028 esrally.utils.process DEBUG Running subprocess [git -C /~/.rally/benchmarks/teams/default --version] with logging.
2024-12-02 07:08:13,485 ActorAddr-(T|:60793)/PID:2028 esrally.utils.process DEBUG git version 2.34.1

2024-12-02 07:08:13,486 ActorAddr-(T|:60793)/PID:2028 esrally.utils.process DEBUG Subprocess [git -C /~/.rally/benchmarks/teams/default --version] finished with return code [0].
2024-12-02 07:08:13,486 ActorAddr-(T|:60793)/PID:2028 esrally.utils.process DEBUG Running subprocess [git -C /~/.rally/benchmarks/teams/default fetch --prune --tags origin] with logging.
2024-12-02 07:08:14,134 ActorAddr-(T|:60793)/PID:2028 esrally.utils.process DEBUG Subprocess [git -C /~/.rally/benchmarks/teams/default fetch --prune --tags origin] finished with return code [0].
2024-12-02 07:08:14,135 ActorAddr-(T|:60793)/PID:2028 esrally.utils.process DEBUG Running subprocess [git -C /~/.rally/benchmarks/teams/default --version] with logging.
2024-12-02 07:08:14,137 ActorAddr-(T|:60793)/PID:2028 esrally.utils.process DEBUG git version 2.34.1

2024-12-02 07:08:14,137 ActorAddr-(T|:60793)/PID:2028 esrally.utils.process DEBUG Subprocess [git -C /~/.rally/benchmarks/teams/default --version] finished with return code [0].
2024-12-02 07:08:14,137 ActorAddr-(T|:60793)/PID:2028 esrally.utils.process DEBUG Running subprocess [git -C /~/.rally/benchmarks/teams/default for-each-ref refs/remotes/ --format='%(refname:short)'] with output.
2024-12-02 07:08:14,140 ActorAddr-(T|:60793)/PID:2028 esrally.utils.repo INFO Checking out [master] in [/~/.rally/benchmarks/teams/default] for distribution version [None].
2024-12-02 07:08:14,140 ActorAddr-(T|:60793)/PID:2028 esrally.utils.process DEBUG Running subprocess [git -C /~/.rally/benchmarks/teams/default --version] with logging.
2024-12-02 07:08:14,142 ActorAddr-(T|:60793)/PID:2028 esrally.utils.process DEBUG git version 2.34.1

2024-12-02 07:08:14,142 ActorAddr-(T|:60793)/PID:2028 esrally.utils.process DEBUG Subprocess [git -C /~/.rally/benchmarks/teams/default --version] finished with return code [0].
2024-12-02 07:08:14,142 ActorAddr-(T|:60793)/PID:2028 esrally.utils.process DEBUG Running subprocess [git -C /~/.rally/benchmarks/teams/default checkout master] with logging.
2024-12-02 07:08:14,147 ActorAddr-(T|:60793)/PID:2028 esrally.utils.process INFO Already on 'master'
Your branch is up to date with 'origin/master'.

2024-12-02 07:08:14,148 ActorAddr-(T|:60793)/PID:2028 esrally.utils.process DEBUG Subprocess [git -C /~/.rally/benchmarks/teams/default checkout master] finished with return code [0].
2024-12-02 07:08:14,148 ActorAddr-(T|:60793)/PID:2028 esrally.utils.repo INFO Rebasing on [master] in [/~/.rally/benchmarks/teams/default] for distribution version [None].
2024-12-02 07:08:14,148 ActorAddr-(T|:60793)/PID:2028 esrally.utils.process DEBUG Running subprocess [git -C /~/.rally/benchmarks/teams/default --version] with logging.
2024-12-02 07:08:14,189 ActorAddr-(T|:60821)/PID:2027 esrally.actor INFO Received a benchmark failure from [ActorAddr-(T|:60793)] and will forward it now.
2024-12-02 07:08:14,150 ActorAddr-(T|:60793)/PID:2028 esrally.utils.process DEBUG git version 2.34.1

2024-12-02 07:08:14,190 -not-actor-/PID:2016 esrally.racecontrol ERROR A benchmark failure has occurred
2024-12-02 07:08:14,190 -not-actor-/PID:2016 esrally.racecontrol INFO Telling benchmark actor to exit.
2024-12-02 07:08:14,150 ActorAddr-(T|:60793)/PID:2028 esrally.utils.process DEBUG Subprocess [git -C /~/.rally/benchmarks/teams/default --version] finished with return code [0].
2024-12-02 07:08:14,150 ActorAddr-(T|:60793)/PID:2028 esrally.utils.process DEBUG Running subprocess [git -C /~/.rally/benchmarks/teams/default --version] with logging.
2024-12-02 07:08:14,191 ActorAddr-(T|:60821)/PID:2027 esrally.actor DEBUG BenchmarkActor received unknown message [ActorExitRequest] (ignoring).
2024-12-02 07:08:14,153 ActorAddr-(T|:60793)/PID:2028 esrally.utils.process DEBUG git version 2.34.1

2024-12-02 07:08:14,193 ActorAddr-(T|:60821)/PID:2027 esrally.actor DEBUG BenchmarkActor received unknown message [ChildActorExited:ActorAddr-(T|:60793)] (ignoring).
2024-12-02 07:08:14,153 ActorAddr-(T|:60793)/PID:2028 esrally.utils.process DEBUG Subprocess [git -C /~/.rally/benchmarks/teams/default --version] finished with return code [0].
2024-12-02 07:08:14,153 ActorAddr-(T|:60793)/PID:2028 esrally.utils.process DEBUG Running subprocess [git -C /~/.rally/benchmarks/teams/default checkout master] with logging.
2024-12-02 07:08:14,160 ActorAddr-(T|:60793)/PID:2028 esrally.utils.process INFO Already on 'master'
Your branch is up to date with 'origin/master'.

2024-12-02 07:08:14,160 ActorAddr-(T|:60793)/PID:2028 esrally.utils.process DEBUG Subprocess [git -C /~/.rally/benchmarks/teams/default checkout master] finished with return code [0].
2024-12-02 07:08:14,160 ActorAddr-(T|:60793)/PID:2028 esrally.utils.process DEBUG Running subprocess [git -C /~/.rally/benchmarks/teams/default rebase origin/master] with logging.
2024-12-02 07:08:14,169 ActorAddr-(T|:60793)/PID:2028 esrally.utils.process INFO Current branch master is up to date.

2024-12-02 07:08:14,170 ActorAddr-(T|:60793)/PID:2028 esrally.utils.process DEBUG Subprocess [git -C /~/.rally/benchmarks/teams/default rebase origin/master] finished with return code [0].
2024-12-02 07:08:14,170 ActorAddr-(T|:60793)/PID:2028 esrally.utils.process DEBUG Running subprocess [git -C /~/.rally/benchmarks/teams/default --version] with logging.
2024-12-02 07:08:14,172 ActorAddr-(T|:60793)/PID:2028 esrally.utils.process DEBUG git version 2.34.1

2024-12-02 07:08:14,172 ActorAddr-(T|:60793)/PID:2028 esrally.utils.process DEBUG Subprocess [git -C /~/.rally/benchmarks/teams/default --version] finished with return code [0].
2024-12-02 07:08:14,172 ActorAddr-(T|:60793)/PID:2028 esrally.utils.process DEBUG Running subprocess [git -C /~/.rally/benchmarks/teams/default rev-parse HEAD] with output.
2024-12-02 07:08:14,176 ActorAddr-(T|:60793)/PID:2028 esrally.actor INFO Cluster consisting of [{'host': 'x.x.x.x', 'port': 9200, 'use_ssl': True}] will be provisioned by Rally.
2024-12-02 07:08:14,176 ActorAddr-(T|:60793)/PID:2028 esrally.actor ERROR Error in mechanic
Traceback (most recent call last):

  File "/~/.local/lib/python3.10/site-packages/esrally/actor.py", line 92, in guard
    return f(self, msg, sender)

  File "/~/.local/lib/python3.10/site-packages/esrally/mechanic/mechanic.py", line 396, in receiveMsg_StartEngine
    self.children = [None] * len(nodes_by_host(to_ip_port(hosts)))

  File "/~/.local/lib/python3.10/site-packages/esrally/mechanic/mechanic.py", line 295, in to_ip_port
    raise exceptions.SystemSetupError(

esrally.exceptions.SystemSetupError: When specifying nodes to be managed by Rally you can only supply hostname:port pairs (e.g. 'localhost:9200'), any additional options cannot be supported.

2024-12-02 07:08:14,192 ActorAddr-(T|:60793)/PID:2028 esrally.actor DEBUG MechanicActor#receiveMessage unrecognized(msg = [<class 'thespian.actors.ActorExitRequest'>] sender = [ActorAddr-(T|:60821)])
2024-12-02 07:08:17,194 -not-actor-/PID:2016 esrally.rally INFO Attempting to shutdown internal actor system.
2024-12-02 07:08:17,196 -not-actor-/PID:2026 root INFO ActorSystem Logging Shutdown
2024-12-02 07:08:17,217 -not-actor-/PID:2025 root INFO ---- Actor System shutdown
2024-12-02 07:08:17,219 -not-actor-/PID:2016 esrally.rally INFO Shutdown completed.
2024-12-02 07:08:17,219 -not-actor-/PID:2016 esrally.rally ERROR Cannot run subcommand [race].
Traceback (most recent call last):
  File "/~/.local/lib/python3.10/site-packages/esrally/rally.py", line 1189, in dispatch_sub_command
    race(cfg, args.kill_running_processes)
  File "/~/.local/lib/python3.10/site-packages/esrally/rally.py", line 937, in race
    with_actor_system(racecontrol.run, cfg)
  File "/~/.local/lib/python3.10/site-packages/esrally/rally.py", line 967, in with_actor_system
    runnable(cfg)
  File "/~/.local/lib/python3.10/site-packages/esrally/racecontrol.py", line 408, in run
    raise e
  File "/~/.local/lib/python3.10/site-packages/esrally/racecontrol.py", line 405, in run
    pipeline(cfg)
  File "/~/.local/lib/python3.10/site-packages/esrally/racecontrol.py", line 74, in __call__
    self.target(cfg)
  File "/~/.local/lib/python3.10/site-packages/esrally/racecontrol.py", line 331, in from_sources
    return race(cfg, sources=True)
  File "/~/.local/lib/python3.10/site-packages/esrally/racecontrol.py", line 302, in race
    raise exceptions.RallyError(result.message, result.cause)
esrally.exceptions.RallyError: Traceback (most recent call last):
  File "/~/.local/lib/python3.10/site-packages/esrally/actor.py", line 92, in guard
    return f(self, msg, sender)
  File "/~/.local/lib/python3.10/site-packages/esrally/mechanic/mechanic.py", line 396, in receiveMsg_StartEngine
    self.children = [None] * len(nodes_by_host(to_ip_port(hosts)))
  File "/~/.local/lib/python3.10/site-packages/esrally/mechanic/mechanic.py", line 295, in to_ip_port
    raise exceptions.SystemSetupError(
esrally.exceptions.SystemSetupError: When specifying nodes to be managed by Rally you can only supply hostname:port pairs (e.g. 'localhost:9200'), any additional options cannot be supported.

My config:

[meta]
config.version = 17

[system]
env.name = dev

[node]
root.dir = /~/.rally/benchmarks
src.root.dir = /~/.rally/benchmarks/src

[source]
remote.repo.url = https://github.com/elastic/elasticsearch.git
elasticsearch.src.subdir = elasticsearch

[benchmarks]
local.dataset.cache = /~/.rally/benchmarks/data

[reporting]
datastore.type = elasticsearch
datastore.host = x.x.x.x
datastore.port = 9200
datastore.secure = true
datastore.user = elastic
datastore.password = changeme
datastore.ssl.verification_mode = none

[tracks]
default.url = https://github.com/elastic/rally-tracks

[teams]
default.url = https://github.com/elastic/rally-teams

[defaults]
preserve_benchmark_candidate = false

[distributions]
release.cache = true
8

I'm glad it worked.

Those configs i got from chatgpt.

ChatGPT was hallucinating. It provided non-existent options. It's best to refer to Rally documentation, and start from the auto-generated configuration file (Rally creates it if it's missing).

What I meant by different is that we recommend datastore.host from [reporting] configuration section to be different than the test target configured via --target-hosts command-line option. Otherwise there is an interference. On one hand Rally creates test load and takes measurements, but in parallel it also ingests those measurements back into Elasticsearch cluster which does not have to be negligible for end results.

also, if i run the command without --pipeline=benchmark-only , i seem to get an error

Thanks for checking. Yes, --pipeline=benchmark-only is required when testing an existing cluster.