When running a race on a 3 node Elasticsearch cluster we are getting the following error. We were able to run a benchmark on a single node cluster.
The single node had a document count of 3,556,667 , whereas the 3 node cluster has a document count of 35,845,521
Version of Rally - 2.8.0
Version of Elastic-Search - 7.17.0
The document count of index - 35,845,521
Followed the instructions in this link
Tips and Tricks - Rally 2.8.0 documentation (esrally.readthedocs.io)
"
[INFO] Race id is [d5179e91-1c44-44a6-9f57-72bcd08891a2]
[ERROR] Cannot race. Error in task executor
Cannot download data because no base URL is provided. "
We tried running the benchmark with the Elastic Search DNS and also with the IP's
esrally race --track=/home/ec2-user/tracks/ES-Cluster --target-hosts=Zentity.DNS.Com:9200 --pipeline=benchmark-only --client-options="basic_auth_user:'user',basic_auth_password:'password'"
esrally race --track=/home/ec2-user/tracks/ES-Cluster --target-hosts=10.149.10.14:9200,10.149.10.248:9200,10.149.11.43:9200 --pipeline=benchmark-only --client-options="basic_auth_user:'user',basic_auth_password:'password'"
{% import "rally.helpers" as rally with context %}
{
"version": 2,
"description": "Tracker-generated track for ES-Cluster",
"indices": [
{
"name": "aps",
"body": "aps.json"
}
],
"corpora": [
{
"name": "aps",
"documents": [
{
"target-index": "aps",
"source-file": "aps-documents.json.bz2",
"document-count": 39695546,
"compressed-bytes": 2144165370,
"uncompressed-bytes": 36715007720
}
]
}
],
"schedule": [
{
"operation": "delete-index"
},
{
"operation": {
"operation-type": "create-index",
"settings": {{index_settings | default({}) | tojson}}
}
},
{
"operation": {
"operation-type": "cluster-health",
"index": "aps",
"request-params": {
"wait_for_status": "{{cluster_health | default('yellow')}}",
"wait_for_no_relocating_shards": "true"
},
"retry-until-success": true
}
},
{
"operation": {
"operation-type": "bulk",
"bulk-size": {{bulk_size | default(5000)}},
"ingest-percentage": {{ingest_percentage | default(100)}}
},
"clients": {{bulk_indexing_clients | default(8)}}
},
{
"operation": {
"operation-type": "search",
"params": {
"search_request": {
"index": "aps",
"body": {
"query": {
"match_all": {}
}
}
}
}
}
}
]
}
[meta]
config.version = 17
[system]
env.name = local
[node]
root.dir = /home/ec2-user/.rally/benchmarks
src.root.dir = /home/ec2-user/.rally/benchmarks/src
[source]
remote.repo.url = https://github.com/elastic/elasticsearch.git
elasticsearch.src.subdir = elasticsearch
[benchmarks]
local.dataset.cache = /home/ec2-user/.rally/benchmarks/data
[reporting]
datastore.type = in-memory
datastore.host =
datastore.port =
datastore.secure = False
datastore.user =
datastore.password =
[tracks]
default.url = https://github.com/elastic/rally-tracks
[teams]
default.url = https://github.com/elastic/rally-teams
[defaults]
preserve_benchmark_candidate = false
[distributions]
release.cache = true
2023-06-28 18:34:02,354 -not-actor-/PID:9670 esrally.rally INFO OS [uname_result(system='Linux', node='', release='3.10.0-1160.92.1.el7.x86_64', version='#1 SMP Thu May 18 11:23:40 UTC 2023', machine='x86_64', processor='x86_64')]
2023-06-28 18:34:02,355 -not-actor-/PID:9670 esrally.rally INFO Python [namespace(_multiarch='x86_64-linux-gnu', cache_tag='cpython-38', hexversion=50860272, name='cpython', version=sys.version_info(major=3, minor=8, micro=16, releaselevel='final', serial=0))]
2023-06-28 18:34:02,355 -not-actor-/PID:9670 esrally.rally INFO Rally version [2.8.0]
2023-06-28 18:34:02,355 -not-actor-/PID:9670 esrally.utils.net INFO Connecting directly to the Internet (no proxy support) for [all_proxy].
2023-06-28 18:34:02,355 -not-actor-/PID:9670 esrally.utils.net INFO Connecting directly to the Internet (no proxy support) for [all_proxy].
2023-06-28 18:34:02,355 -not-actor-/PID:9670 esrally.rally INFO Cleaning track dependency directory [/home/ec2-user/.rally/libs]...
2023-06-28 18:34:02,385 -not-actor-/PID:9670 esrally.rally INFO Actor system already running locally? [False]
2023-06-28 18:34:02,385 -not-actor-/PID:9670 esrally.actor INFO Starting actor system with system base [multiprocTCPBase] and capabilities [{'coordinator': True, 'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900'}].
2023-06-28 18:34:02,526 -not-actor-/PID:9685 root INFO ++++ Actor System gen (3, 10) started, admin @ ActorAddr-(T|:1900)
2023-06-28 18:34:02,536 -not-actor-/PID:9670 esrally.racecontrol INFO Race id is [a2b51c7a-a4ae-474a-a76f-9ac3951e35b3]
2023-06-28 18:34:02,536 -not-actor-/PID:9670 esrally.racecontrol INFO User specified pipeline [benchmark-only].
2023-06-28 18:34:02,537 -not-actor-/PID:9670 esrally.racecontrol INFO Using configured hosts [{'host': 'zentity-xx.xx.xx.local', 'port': 9200}]
2023-06-28 18:34:02,539 ActorAddr-(T|:1900)/PID:9685 esrally.actor DEBUG Capabilities [{'coordinator': True, 'ip': '127.0.0.1', 'Convention Address.IPv4': '127.0.0.1:1900', 'Thespian ActorSystem Name': 'multiprocTCPBase', 'Thespian ActorSystem Version': 2, 'Thespian Watch Supported': True, 'Python Version': (3, 8, 16, 'final', 0), 'Thespian Generation': (3, 10), 'Thespian Version': '1687977242510'}] match requirements [{'coordinator': True}].
2023-06-28 18:34:02,547 ActorAddr-(T|:41812)/PID:9687 esrally.client.factory INFO Creating ES client connected to [{'host': 'zentity-xx.xx.xx.local', 'port': 9200}] with options [{'timeout': 60, 'basic_auth_user': 'user', 'basic_auth_password': '*****'}]
2023-06-28 18:34:03,114 ActorAddr-(T|:41812)/PID:9687 esrally.racecontrol INFO Automatically derived distribution version [7.17.0]
2023-06-28 18:34:03,897 ActorAddr-(T|:41812)/PID:9687 esrally.utils.process INFO From https://github.com/elastic/rally-tracks
336c149..64fa238 master -> origin/master
336c149..64fa238 8.7 -> origin/8.7
2023-06-28 18:34:03,946 ActorAddr-(T|:41812)/PID:9687 esrally.utils.repo INFO Checking out [7.17] in [/home/ec2-user/.rally/benchmarks/tracks/default] for distribution version [7.17.0].
2023-06-28 18:34:04,147 ActorAddr-(T|:41812)/PID:9687 esrally.utils.process INFO Switched to branch '7.17'
Your branch is up-to-date with 'origin/7.17'.
2023-06-28 18:34:04,148 ActorAddr-(T|:41812)/PID:9687 esrally.utils.repo INFO Rebasing on [7.17] in [/home/ec2-user/.rally/benchmarks/tracks/default] for distribution version [7.17.0].
2023-06-28 18:34:05,397 ActorAddr-(T|:41812)/PID:9687 esrally.utils.process INFO Already on '7.17'
Your branch is up-to-date with 'origin/7.17'.
2023-06-28 18:34:05,849 ActorAddr-(T|:41812)/PID:9687 esrally.utils.process INFO Current branch 7.17 is up to date.
2023-06-28 18:34:05,948 ActorAddr-(T|:43559)/PID:9758 esrally.actor INFO Received signal from race control to start engine.
2023-06-28 18:34:05,902 ActorAddr-(T|:41812)/PID:9687 esrally.track.loader INFO Reading track specification file [/home/ec2-user/tracks/ES-Cluster/track.json].
2023-06-28 18:34:05,949 ActorAddr-(T|:43559)/PID:9758 esrally.actor INFO Cluster will not be provisioned by Rally.
2023-06-28 18:34:05,920 ActorAddr-(T|:41812)/PID:9687 esrally.track.loader INFO Final rendered track for '/home/ec2-user/tracks/ES-Cluster/track.json' has been written to '/tmp/tmp5n6v0rtl.json'.
2023-06-28 18:34:05,932 ActorAddr-(T|:41812)/PID:9687 esrally.track.loader INFO Loading template [definition for index aps in aps.json].
2023-06-28 18:34:05,937 ActorAddr-(T|:41812)/PID:9687 esrally.metrics INFO Creating in-memory metrics store
2023-06-28 18:34:05,937 ActorAddr-(T|:41812)/PID:9687 esrally.metrics INFO Opening metrics store for race timestamp=[20230628T183402Z], track=[/home/ec2-user/tracks/ES-Cluster], challenge=[default], car=[['external']]
2023-06-28 18:34:05,938 ActorAddr-(T|:41812)/PID:9687 esrally.metrics INFO Creating file race store
2023-06-28 18:34:05,938 ActorAddr-(T|:41812)/PID:9687 esrally.actor INFO Asking mechanic to start the engine.
2023-06-28 18:34:05,950 ActorAddr-(T|:41812)/PID:9687 esrally.actor INFO Mechanic has started engine successfully.
2023-06-28 18:34:05,951 ActorAddr-(T|:41812)/PID:9687 esrally.actor INFO Telling driver to prepare for benchmarking.
2023-06-28 18:34:05,958 ActorAddr-(T|:37695)/PID:9759 esrally.metrics INFO Creating in-memory metrics store
2023-06-28 18:34:05,958 ActorAddr-(T|:37695)/PID:9759 esrally.metrics INFO Opening metrics store for race timestamp=[20230628T183402Z], track=[/home/ec2-user/tracks/ES-Cluster], challenge=[default], car=[['external']]
2023-06-28 18:34:05,959 ActorAddr-(T|:37695)/PID:9759 esrally.client.factory INFO Creating ES client connected to [{'host': 'zentity-xx.xx.xx.local', 'port': 9200}] with options [{'timeout': 60, 'basic_auth_user': 'user', 'basic_auth_password': '*****','retry_on_timeout': True}]
2023-06-28 18:34:05,960 ActorAddr-(T|:37695)/PID:9759 esrally.driver.driver INFO Checking if REST API is available.
2023-06-28 18:34:05,967 ActorAddr-(T|:37695)/PID:9759 esrally.driver.driver INFO REST API is available.
2023-06-28 18:34:05,969 ActorAddr-(T|:37695)/PID:9759 esrally.actor INFO Starting prepare track process on hosts [['localhost']]
2023-06-28 18:34:05,975 ActorAddr-(T|:40599)/PID:9760 esrally.actor INFO Track Preparator started
2023-06-28 18:34:06,351 ActorAddr-(T|:40599)/PID:9760 esrally.track.loader INFO Reading track specification file [/home/ec2-user/tracks/ES-Cluster/track.json].
2023-06-28 18:34:06,363 ActorAddr-(T|:40599)/PID:9760 esrally.track.loader INFO Final rendered track for '/home/ec2-user/tracks/ES-Cluster/track.json' has been written to '/tmp/tmppqm07b7b.json'.
2023-06-28 18:34:06,370 ActorAddr-(T|:40599)/PID:9760 esrally.track.loader INFO Loading template [definition for index aps in aps.json].
2023-06-28 18:34:06,377 ActorAddr-(T|:40599)/PID:9760 esrally.actor INFO Preparing track [/home/ec2-user/tracks/ES-Cluster]
2023-06-28 18:34:06,377 ActorAddr-(T|:40599)/PID:9760 esrally.actor INFO Reloading track [/home/ec2-user/tracks/ES-Cluster] to ensure plugins are up-to-date.
2023-06-28 18:34:06,805 ActorAddr-(T|:40599)/PID:9760 esrally.track.loader INFO Reading track specification file [/home/ec2-user/tracks/ES-Cluster/track.json].
2023-06-28 18:34:06,817 ActorAddr-(T|:40599)/PID:9760 esrally.track.loader INFO Final rendered track for '/home/ec2-user/tracks/ES-Cluster/track.json' has been written to '/tmp/tmpud4p29hq.json'.
2023-06-28 18:34:06,824 ActorAddr-(T|:40599)/PID:9760 esrally.track.loader INFO Loading template [definition for index aps in aps.json].
2023-06-28 18:34:17,279 ActorAddr-(T|:44913)/PID:9791 esrally.track.loader INFO Resolved data root directory for document corpus [aps] intrack [/home/ec2-user/tracks/ES-Cluster] to [['/home/ec2-user/.rally/benchmarks/data/aps']].
2023-06-28 18:34:22,284 ActorAddr-(T|:44913)/PID:9791 esrally.driver.driver ERROR Worker failed. Notifying parent...
Traceback (most recent call last):
File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/home/ec2-user/.local/lib/python3.8/site-packages/esrally/track/loader.py", line 457, in prepare_docs
preparator.prepare_document_set(document_set, data_root[0])
File "/home/ec2-user/.local/lib/python3.8/site-packages/esrally/track/loader.py", line 620, in prepare_document_set
self.downloader.download(document_set.base_url, target_path, expected_size)
File "/home/ec2-user/.local/lib/python3.8/site-packages/esrally/track/loader.py", line 508, in download
raise exceptions.DataError("Cannot download data because no base URL is provided.")
esrally.exceptions.DataError: Cannot download data because no base URL is provided.
2023-06-28 18:34:22,303 ActorAddr-(T|:37695)/PID:9759 esrally.actor ERROR Main driver received a fatal exception from a load generator. Shutting down.
2023-06-28 18:34:22,303 ActorAddr-(T|:37695)/PID:9759 esrally.metrics INFO Closing metrics store.
2023-06-28 18:34:22,304 ActorAddr-(T|:41812)/PID:9687 esrally.actor INFO Received a benchmark failure from [ActorAddr-(T|:37695)] and will forward it now.
2023-06-28 18:34:22,306 -not-actor-/PID:9670 esrally.racecontrol ERROR A benchmark failure has occurred
2023-06-28 18:34:22,306 -not-actor-/PID:9670 esrally.racecontrol INFO Telling benchmark actor to exit.
2023-06-28 18:34:22,307 ActorAddr-(T|:37695)/PID:9759 esrally.actor INFO Main driver received ActorExitRequest and will terminate all load generators.
2023-06-28 18:34:25,308 -not-actor-/PID:9670 esrally.rally INFO Attempting to shutdown internal actor system.
2023-06-28 18:34:25,310 -not-actor-/PID:9686 root INFO ActorSystem Logging Shutdown
2023-06-28 18:34:25,330 -not-actor-/PID:9685 root INFO ---- Actor System shutdown
2023-06-28 18:34:25,331 -not-actor-/PID:9670 esrally.rally INFO Actor system is still running. Waiting...
2023-06-28 18:34:26,332 -not-actor-/PID:9670 esrally.rally INFO Shutdown completed.
2023-06-28 18:34:26,332 -not-actor-/PID:9670 esrally.rally ERROR Cannot run subcommand [race].
Traceback (most recent call last):
File "/home/ec2-user/.local/lib/python3.8/site-packages/esrally/rally.py", line 1172, in dispatch_sub_command
race(cfg, args.kill_running_processes)
File "/home/ec2-user/.local/lib/python3.8/site-packages/esrally/rally.py", line 920, in race
with_actor_system(racecontrol.run, cfg)
File "/home/ec2-user/.local/lib/python3.8/site-packages/esrally/rally.py", line 950, in with_actor_system
runnable(cfg)
File "/home/ec2-user/.local/lib/python3.8/site-packages/esrally/racecontrol.py", line 372, in run
raise e
File "/home/ec2-user/.local/lib/python3.8/site-packages/esrally/racecontrol.py", line 369, in run
pipeline(cfg)
File "/home/ec2-user/.local/lib/python3.8/site-packages/esrally/racecontrol.py", line 71, in __call__
self.target(cfg)
File "/home/ec2-user/.local/lib/python3.8/site-packages/esrally/racecontrol.py", line 308, in benchmark_only
return race(cfg, external=True)
File "/home/ec2-user/.local/lib/python3.8/site-packages/esrally/racecontrol.py", line 266, in race
raise exceptions.RallyError(result.message, result.cause)
esrally.exceptions.RallyError: Error in task executor