Cannot download ES distribution 6.6.0 as rally builds incorrect download path

Hi,

I am a Rally newbie and having a go at creating some custom tracks and cars. My OS is Ubuntu Linux 18.04.1 64-bit. Rally version is 1.4.1. I am attempting to align my testing with ES v6.6.0 for reasons I won't go into, although I realise it's a year or two old.

So far, I am able to list my new cars.

Summary
spayne:~ $ esrally list cars --team-repository ~/elasticsearch-src/teams

    ____        ____
   / __ \____ _/ / /_  __
  / /_/ / __ `/ / / / / /
 / _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
                /____/

Available cars:

Name                     Type    Description
-----------------------  ------  --------------------------------------------------------
16gheap                  car     Sets the Java heap to 16GB
1gheap                   car     Sets the Java heap to 1GB
24gheap                  car     Sets the Java heap to 24GB
2gheap                   car     Sets the Java heap to 2GB
4gheap                   car     Sets the Java heap to 4GB
8gheap                   car     Sets the Java heap to 8GB
defaults                 car     Sets the Java heap to 1GB
java-bridge-debug        car     SOME DESCRIPTION
java-no-debug            car     SOME DESCRIPTION
java-standard-debug      car     SOME DESCRIPTION
basic-license            mixin   Basic License
debug-non-safepoints     mixin   More accurate CPU profiles
ea                       mixin   Enables Java assertions
fp                       mixin   Preserves frame pointers
g1gc                     mixin   Enables the G1 garbage collector
trial-license            mixin   Trial License
unpooled                 mixin   Enables Netty's unpooled allocator
x-pack-ml                mixin   X-Pack Machine Learning
x-pack-monitoring-http   mixin   X-Pack Monitoring (HTTP exporter)
x-pack-monitoring-local  mixin   X-Pack Monitoring (local exporter)
x-pack-security          mixin   X-Pack Security

-------------------------------
[INFO] SUCCESS (took 0 seconds)
-------------------------------

and tracks:

Summary
spayne:~ $ esrally list tracks --track-path=~/elasticsearch-src/custom-tracks

    ____        ____
   / __ \____ _/ / /_  __
  / /_/ / __ `/ / / / / /
 / _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
                /____/

Available tracks:

Name           Description                   Documents    Compressed Size    Uncompressed Size
-------------  ----------------------------  -----------  -----------------  -------------------
custom-tracks  Tutorial benchmark for Rally  11,658,903   N/A                1.4 GB

-------------------------------
[INFO] SUCCESS (took 0 seconds)
-------------------------------

However when I try to run the benchmark I get an error.
When I run this command from the console:

esrally --car=java-no-debug --distribution-version=6.6.0 --track-path=~/elasticsearch-src/custom-tracks --team-path=~/elasticsearch-src/team

it gives an error:

Summary
   ____        ____
   / __ \____ _/ / /_  __
  / /_/ / __ `/ / / / / /
 / _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
                /____/

[INFO] Preparing for race ...
[ERROR] Cannot race. ('Cannot download Elasticsearch distribution from [https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-oss-6.6.0-linux-x86_64.tar.gz]. Please check that the specified version [6.6.0] is correct.', None)
	Traceback (most recent call last):
  File "/home/spayne/.local/lib/python3.6/site-packages/esrally/mechanic/supplier.py", line 355, in fetch
    net.download(download_url, distribution_path, progress_indicator=progress)
  File "/home/spayne/.local/lib/python3.6/site-packages/esrally/utils/net.py", line 152, in download
    expected_size_in_bytes = download_http(url, tmp_data_set_path, expected_size_in_bytes, progress_indicator)
  File "/home/spayne/.local/lib/python3.6/site-packages/esrally/utils/net.py", line 116, in download_http
    raise urllib.error.HTTPError(url, r.status, "", None, None)
urllib.error.HTTPError: HTTP Error 404: 

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/spayne/.local/lib/python3.6/site-packages/esrally/mechanic/mechanic.py", line 565, in receiveMsg_StartNodes
    self.mechanic.start_engine()
  File "/home/spayne/.local/lib/python3.6/site-packages/esrally/mechanic/mechanic.py", line 676, in start_engine
    binaries = self.supply()
  File "/home/spayne/.local/lib/python3.6/site-packages/esrally/mechanic/supplier.py", line 218, in __call__
    supplier.fetch()
  File "/home/spayne/.local/lib/python3.6/site-packages/esrally/mechanic/supplier.py", line 361, in fetch
    "version [%s] is correct." % (download_url, self.version))
esrally.exceptions.SystemSetupError: ('Cannot download Elasticsearch distribution from [https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-oss-6.6.0-linux-x86_64.tar.gz]. Please check that the specified version [6.6.0] is correct.', None)

From checking the available past releases, there is one similar:
Available:
https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-oss-6.6.0.tar.gz
My stuff wants:
https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-oss-6.6.0-linux-x86_64.tar.gz

Is it possible to pass a modified distribution-version param to locate the correct remote release version and fix this?

Alternatively (even better), is it possible for me to run this with the same ES distro that I had already downloaded to my machine to use "for real" rather than pull another one down just to run the rally? I'd prefer that to using a separate version as it's the one I am focused on.

Or is there something broken on the download site?

Please advise, thanks.

Simon

P.S. my rally.ini file:

[meta]
config.version = 17

[system]
env.name = local

[node]
root.dir = /home/spayne/.rally/benchmarks
src.root.dir = /home/spayne/elasticsearch-src

[source]
remote.repo.url = https://github.com/elastic/elasticsearch.git
elasticsearch.src.subdir = elasticsearch

[benchmarks]
local.dataset.cache = /home/spayne/.rally/benchmarks/data

[reporting]
datastore.type = in-memory
datastore.host = 
datastore.port = 
datastore.secure = False
datastore.user = 
datastore.password = 

[tracks]
default.url = https://github.com/elastic/rally-tracks

[teams]
default.url = https://github.com/elastic/rally-teams

[defaults]
preserve_benchmark_candidate = False

[distributions]
release.cache = true

and rally.log:

Summary
(truncated to last few lines)
2020-02-27 11:09:14,926 ActorAddr-(T|:38069)/PID:5198 esrally.metrics INFO Creating file race store
2020-02-27 11:09:14,926 ActorAddr-(T|:38069)/PID:5198 esrally.mechanic.mechanic WARNING Cannot store system metrics: ('No race with race id [b50d6f7e-a699-4d8d-a894-4dd27713b510]', None).
2020-02-27 11:09:14,926 ActorAddr-(T|:38069)/PID:5198 esrally.metrics INFO Closing metrics store.
2020-02-27 11:09:14,928 ActorAddr-(T|:33407)/PID:5196 esrally.actor ERROR Child actor exited with [ChildActorExited:ActorAddr-(T|:41233)] while in status [starting].
2020-02-27 11:09:14,943 ActorAddr-(T|:34751)/PID:5195 esrally.actor INFO BenchmarkActor received unknown message [ChildActorExited:ActorAddr-(T|:33407)] (ignoring).
2020-02-27 11:09:17,926 -not-actor-/PID:5186 esrally.rally INFO Attempting to shutdown internal actor system.
2020-02-27 11:09:17,928 -not-actor-/PID:5194 root INFO ActorSystem Logging Shutdown
2020-02-27 11:09:17,949 -not-actor-/PID:5193 root INFO ---- Actor System shutdown
2020-02-27 11:09:17,950 -not-actor-/PID:5186 esrally.rally INFO Actor system is still running. Waiting...
2020-02-27 11:09:18,952 -not-actor-/PID:5186 esrally.rally INFO Shutdown completed.
2020-02-27 11:09:18,953 -not-actor-/PID:5186 esrally.rally ERROR Cannot run subcommand [race].
Traceback (most recent call last):
  File "/home/spayne/.local/lib/python3.6/site-packages/esrally/rally.py", line 714, in dispatch_sub_command
    race(cfg)
  File "/home/spayne/.local/lib/python3.6/site-packages/esrally/rally.py", line 624, in race
    with_actor_system(racecontrol.run, cfg)
  File "/home/spayne/.local/lib/python3.6/site-packages/esrally/rally.py", line 652, in with_actor_system
    runnable(cfg)
  File "/home/spayne/.local/lib/python3.6/site-packages/esrally/racecontrol.py", line 365, in run
    raise e
  File "/home/spayne/.local/lib/python3.6/site-packages/esrally/racecontrol.py", line 362, in run
    pipeline(cfg)
  File "/home/spayne/.local/lib/python3.6/site-packages/esrally/racecontrol.py", line 79, in __call__
    self.target(cfg)
  File "/home/spayne/.local/lib/python3.6/site-packages/esrally/racecontrol.py", line 290, in from_distribution
    return race(cfg, distribution=True)
  File "/home/spayne/.local/lib/python3.6/site-packages/esrally/racecontrol.py", line 250, in race
    raise exceptions.RallyError(result.message, result.cause)
esrally.exceptions.RallyError: (Cannot download Elasticsearch distribution from [https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-oss-6.6.0-linux-x86_64.tar.gz]. Please check that the specified version [6.6.0] is correct., 'Traceback (most recent call last):\n  File "/home/spayne/.local/lib/python3.6/site-packages/esrally/mechanic/supplier.py", line 360, in fetch\n    net.download(download_url, distribution_path, progress_indicator=progress)\n  File "/home/spayne/.local/lib/python3.6/site-packages/esrally/utils/net.py", line 152, in download\n    expected_size_in_bytes = download_http(url, tmp_data_set_path, expected_size_in_bytes, progress_indicator)\n  File "/home/spayne/.local/lib/python3.6/site-packages/esrally/utils/net.py", line 116, in download_http\n    raise urllib.error.HTTPError(url, r.status, "", None, None)\nurllib.error.HTTPError: HTTP Error 404: \n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  File "/home/spayne/.local/lib/python3.6/site-packages/esrally/mechanic/mechanic.py", line 565, in receiveMsg_StartNodes\n    self.mechanic.start_engine()\n  File "/home/spayne/.local/lib/python3.6/site-packages/esrally/mechanic/mechanic.py", line 676, in start_engine\n    binaries = self.supply()\n  File "/home/spayne/.local/lib/python3.6/site-packages/esrally/mechanic/supplier.py", line 218, in __call__\n    supplier.fetch()\n  File "/home/spayne/.local/lib/python3.6/site-packages/esrally/mechanic/supplier.py", line 366, in fetch\n    "version [%s] is correct." % (download_url, self.version))\nesrally.exceptions.SystemSetupError: (\'Cannot download Elasticsearch distribution from [https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-oss-6.6.0-linux-x86_64.tar.gz]. Please check that the specified version [6.6.0] is correct.\', None)\n')

Ahah! I found that the URL template was configured in my $TEAM_PATH/v1/vanilla/config.ini where $TEAM_PATH =~/elasticsearch-src/teams. This line in particular:

jdk.bundled.release_url = https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-oss-{{VERSION}}-{{OSNAME}}-{{ARCH}}.tar.gz

when I changed it to remove the -{{OSNAME}}-{{ARCH}} the download worked.

Phew! On to the next challenge...

1 Like

Great sleuthing @sipayne! When in doubt, test out a rally run without any external additions to see if it also fails. I typically grab the generic "run a race" command from our docs. Feel free to ask us if you have any other questions and welcome to the community.

1 Like

Hi,

just as an additional info: Is it possible that you cloned https://github.com/elastic/rally-teams and based your custom team configuration on the master branch of that repo? We have a versioning scheme to support a wide range of Elasticsearch versions and if you look in branch 6, you'll notice that the download URL is correct for version 6.x:

# deprecated
release_url = https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-oss-{{VERSION}}.tar.gz
# new
jdk.unbundled.release_url = https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-oss-{{VERSION}}.tar.gz

Daniel

Hi @danielmitterdorfer,

Actually I didn't know about rally-teams, thank you for letting me know, it might have saved me some struggles. I copied .rally/benchmarks/teams/default but since that is a git repo it has its own versioning, and I may have started from the master branch rather than branch 6. I'm a bit hazy in my recollection what I did!

Simon

Hi Simon,

This is actually your local checkout of GitHub - elastic/rally-teams: Default Elasticsearch configurations for the Elasticsearch benchmarking tool Rally. Our docs about so-called team repositories explain their usage in a bit more detail.

Daniel

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.