The below bugs appers when running esrally

Elastic Stack Elasticsearch
1

[root@laoyang .rally]# esrally
fatal: Not a git repository (or any of the parent directories): .git

____        ____

/ __ ____ / / / __
/ // / __ `/ / / / / /
/ , / // / / / // /
// ||_,///_, /
/____/

Writing additional logs to /root/.rally/benchmarks/races/2016-08-03-02-42-21/local/logs/rally_out.log

fatal: Not a git repository (or any of the parent directories): .git
Downloading data from http://benchmarks.elasticsearch.org.s3.amazonaws.com/corpora/geonames/documents.json.bz2 (189 MB)

The above information will last for a long time, at the end, tips: can not download....

The detail info from /root/.rally/benchmarks/races/2016-08-03-02-42-21/local/logs/rally_out.log is as below:
2016-08-03 02:42:23,270 rally.track INFO Downloading from [http://benchmarks.elasticsearch.org.s3.amazonaws.com/corpora/geonames/documents.json.bz2] to [/root/.rally/be nchmarks/data/geonames/documents.json.bz2].
268 2016-08-03 03:05:51,298 root ERROR A fatal error occurred while running subcommand [race].
269 Traceback (most recent call last):
270 File "/usr/local/python3/lib/python3.5/site-packages/esrally/rally.py", line 299, in dispatch_sub_command
271 racecontrol.run(cfg)
272 File "/usr/local/python3/lib/python3.5/site-packages/esrally/racecontrol.py", line 336, in run
273 pipeline()
274 File "/usr/local/python3/lib/python3.5/site-packages/esrally/racecontrol.py", line 58, in call
275 step()
276 File "/usr/local/python3/lib/python3.5/site-packages/esrally/racecontrol.py", line 27, in call
277 self.command(self.ctx)
278 File "/usr/local/python3/lib/python3.5/site-packages/esrally/racecontrol.py", line 94, in prepare_track
279 track.prepare_track(ctx.track, ctx.config)
280 File "/usr/local/python3/lib/python3.5/site-packages/esrally/track.py", line 298, in prepare_track
281 download(cfg, data_url, type.document_archive, type.compressed_size_in_bytes)
282 File "/usr/local/python3/lib/python3.5/site-packages/esrally/track.py", line 268, in download
283 net.download(url, local_path, size_in_bytes)
284 File "/usr/local/python3/lib/python3.5/site-packages/esrally/utils/net.py", line 19, in download
285 download_via_http(url, local_path, expected_size_in_bytes)
286 File "/usr/local/python3/lib/python3.5/site-packages/esrally/utils/net.py", line 35, in download_via_http
287 urllib.request.urlretrieve(url, tmp_data_set_path)
288 File "/usr/local/python3/lib/python3.5/urllib/request.py", line 216, in urlretrieve
289 block = fp.read(bs)
290 File "/usr/local/python3/lib/python3.5/http/client.py", line 433, in read
291 n = self.readinto(b)
292 File "/usr/local/python3/lib/python3.5/http/client.py", line 473, in readinto
293 n = self.fp.readinto(b)
294 File "/usr/local/python3/lib/python3.5/socket.py", line 575, in readinto
295 return self._sock.recv_into(b)

I don't know why downloading data from amazonaws.com?
I also don't know the target of documents.json.bz2?

2

Hi @laoyang360,

what bugs me is that the very first line says:

fatal: Not a git repository (or any of the parent directories): .git

We use git for checking out the Elasticsearch source code and for managing our "tracks" (i.e. benchmark descriptions). To me this looks like something in your .rally directory is inconsistent. Rally assumes that it manages this directory all by itself. Did you change anything manually in this directory or did you experience a fatal error previously?

It seems that you did not paste the full log file. Can you please do that?. Can you also share the contents of /root/.rally/rally.ini?

I don't know why downloading data from amazonaws.com?

Rally has to use some data for running a benchmark. And this is the location where it downloads them for you (it does this only once and caches the file locally then).

It looks like you cannot reach the server. Are you maybe behind some proxy?

You can check whether you'd be able to download the file from the machine by issuing a HEAD request with curl:

curl -I http://benchmarks.elasticsearch.org.s3.amazonaws.com/corpora/geonames/documents.json.bz2

This should return something like:

HTTP/1.1 200 OK
x-amz-id-2: 0kwgzT633TbtdWKuEWQz/QdqcIAvmmiEENH5pXOYtFGrsYB2TXyuiXgD7j7SUA1kqDv6BKiA4vo=
x-amz-request-id: 91ECF2E5AF7BA5D6
Date: Wed, 03 Aug 2016 04:56:44 GMT
Last-Modified: Fri, 04 Dec 2015 10:00:49 GMT
ETag: "ab8d5133b42e89e1ddffa281eaff971c-13"
x-amz-meta-s3cmd-attrs: uid:1000/gname:dialout/uname:mike/gid:20/mode:33204/mtime:1426483472/atime:1426472160/md5:d67839c9f3434b6857d32e526c6c01e2/ctime:1426485668
Accept-Ranges: bytes
Content-Type: application/json
Content-Length: 197857614
Server: AmazonS3

If this does not work then you should check how to resolve the network issue.

I also don't know the target of documents.json.bz2?

What do you mean by that? Rally manages all file handling for you. You should not need to care where this file is stored etc.

Daniel

3

@danielmitterdorfer,tks.

The total configuration of my rally.ini is:
[root@laoyang .rally]# cat rally.ini

[meta]
config.version = 5

[system]
root.dir = /root/.rally/benchmarks
log.root.dir = logs
env.name = local

[source]
local.src.dir=/usr/local/elasticsearch
remote.repo.url=https://github.com/elastic/elasticsearch.git

[provisioning]
local.install.dir = install

[runtime]
java8.home = /opt/jdk1.8.0_91

[benchmarks]
local.dataset.cache = ${system:root.dir}/data

[reporting]
datastore.type = elaticsearch
datastore.host = 1xx.xx.xx.xx

datastore.port = 9200
datastore.secure = False
datastore.user =
datastore.password =

[tracks]
default.url = https://github.com/elastic/rally-tracks

When I running esrally, the below information appers:
esrally
fatal: Not a git repository (or any of the parent directories): .git

____        ____

/ __ ____ / / / __
/ // / __ `/ / / / / /
/ , / // / / / // /
// ||_,///_, /
/____/

Writing additional logs to /root/.rally/benchmarks/races/2016-08-03-05-11-27/local/logs/rally_out.log

fatal: Not a git repository (or any of the parent directories): .git
Downloading data from http://benchmarks.elasticsearch.org.s3.amazonaws.com/corpora/geonames/documents.json.bz2 (189 MB) ...

I refer to your tips running:
[root@laoyang yangcy]# curl -I http://benchmarks.elasticsearch.org.s3.amazonaws.com/corpora/geonames/documents.json.bz2
HTTP/1.1 200 OK
x-amz-id-2: qmQBCRfzSWKbetCeE/wmuYFtioHGwaA8gOWrq5m4rpUgrHffDaVpWedPQg9eCj8K
x-amz-request-id: 9E7BFDFDFE168464
Date: Wed, 03 Aug 2016 05:21:37 GMT
Last-Modified: Fri, 04 Dec 2015 10:00:49 GMT
ETag: "ab8d5133b42e89e1ddffa281eaff971c-13"
x-amz-meta-s3cmd-attrs: uid:1000/gname:dialout/uname:mike/gid:20/mode:33204/mtime:1426483472/atime:1426472160/md5:d67839c9f3434b6857d32e526c6c01e2/ctime:1426485668
Accept-Ranges: bytes
Content-Type: application/json
Content-Length: 197857614
Server: AmazonS3

the return result is same as your result.

But I don't know why it download for such long long time?

4

Hi @laoyang360,

oh, I sense we're coming close to finding out what's going on with git. I assume you have already installed an Elasticsearch distribution in /usr/local/elasticsearch?

If yes, Rally expects the Elasticsearch source code in this directory so this is why it won't work. Rally can also automatically manage this directory for you, so there is no need to create anything.

Normally, I do not encourage to change Rally's config file manually but can you change it to some different directory?

I suggest:

local.src.dir=/root/.rally/src

This is what Rally uses as default. You don't have to take care of anything, Rally will automatically create the folder, checkout the sources etc.. Just change the value in the ini file.

But I don't know why it download for such long long time?

The output looks good. I have no clue either why it is taking so long. Are you on a very slow internet connection?

Again, a workaround (usually not needed and not encouraged but I want to help you to move on):

  1. Download the file manually.
  2. Move it to /root/.rally/benchmarks/data/geonames/ and create any folders that don't exist so far

This is what Rally would do for you.

If you download the file manually, please check the download speed and share the results. I'd be really interested.

I hope that gets you going.

It would be great if you could provide more detail about your environment (especially what's going on with your network). Maybe I am able to make Rally more robust for scenarios like yours then because this is not ideal...

Daniel

5

@danielmitterdorfer, tks.
I already changed the ini configuration refer to your said.
Now I download the document.json.bz2 file, My machine and network information are as follow:

system:
[root@laoyang .rally]# uname -a
Linux laoyang 2.6.32-358.el6.x86_64 #1 SMP Fri Feb 22 00:31:26 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux

[root@laoyang .rally]# cat /etc/issue
CentOS release 6.8 (Final)
Kernel \r on an \m

cpu:
processor : 16
vendor_id : AuthenticAMD
cpu family : 16

memory:
[root@laoyang .rally]# cat /proc/meminfo
MemTotal: 32873764 kB
MemFree: 27771572 kB
Buffers: 237444 kB
Cached: 3660128 kB

disk:
[root@laoyang .rally]# df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_root-lv_root
442G 3.8G 415G 1% /
tmpfs 16G 0 16G 0% /dev/shm
/dev/sda1 504M 73M 407M 16% /boot

average network speed:
download document.json.bz2: 130KB/s

When I fixed the bugs or find the new problem, I will tell you later.

6

Hi,

the hardware looks pretty decent. Download speed is a bit low but it should still work. Maybe this is "just" a network timeout issue. When I find time I'll try whether I can reproduce it and make Rally more robust.

Daniel

7

Now, the download skipped refer to what your said above, but the below bug appers.
Do you know the reason? Tks.

[root@laoyang .rally]# esrally
fatal: Not a git repository (or any of the parent directories): .git

____        ____

/ __ ____ / / / __
/ // / __ `/ / / / / /
/ , / // / / / // /
// ||_,///_, /
/____/

Writing additional logs to /root/.rally/benchmarks/races/2016-08-04-00-29-08/local/logs/rally_out.log

fatal: Not a git repository (or any of the parent directories): .git
Preparing for race (might take a few moments) ...

FATAL: Cannot race

Reason: No value for mandatory configuration: section='build', key='gradle.bin'

Please check the log file [/root/.rally/benchmarks/races/2016-08-04-00-29-08/local/logs/rally_out.log] for further details first.

If you need further help, please check Rally's docs at https://esrally.readthedocs.io or ask a question in the forum at https://discuss.elastic.co/c/elasticsearch.

If you think this is a bug, please file a report at https://github.com/elastic/rally/issues and include the log file for this race (/root/.rally/benchmarks/races/2016-08-04-00-29-08/local/logs/rally_out.log).
[root@laoyang .rally]# [/root/.rally/benchmarks/races/2016-08-04-00-29-08/local/logs/rally_out.log/root/.rally/benchmarks/races/2016-08-04-00-29-08/local/logs/rally_out.log
-bash: [/root/.rally/benchmarks/races/2016-08-04-00-29-08/local/logs/rally_out.log/root/.rally/benchmarks/races/2016-08-04-00-29-08/local/logs/rally_out.log: No such file or directory

2016-08-04 00:29:11,482 rally.track INFO [/root/.rally/benchmarks/data/geonames/documents.json.bz2] already exists locally. Skipping download.
268 2016-08-04 00:29:11,482 rally.metrics INFO Creating in-memory race store
269 2016-08-04 00:29:11,483 rally.supplier INFO Skip fetching sources
270 2016-08-04 00:29:11,493 rally.supplier INFO Specified revision [current] on command line results in git revision [046abdc
271 ]
272 2016-08-04 00:29:11,495 root ERROR A fatal error occurred while running subcommand [race].
273 Traceback (most recent call last):
274 File "/usr/local/python3/lib/python3.5/site-packages/esrally/config.py", line 109, in opts
275 return self._opts[self._k(scope, section, key)]
276 KeyError: 'build::gradle.bin'
277
278 During handling of the above exception, another exception occurred:
279
280 Traceback (most recent call last):
281 File "/usr/local/python3/lib/python3.5/site-packages/esrally/rally.py", line 299, in dispatch_sub_command
282 racecontrol.run(cfg)
283 File "/usr/local/python3/lib/python3.5/site-packages/esrally/racecontrol.py", line 336, in run
284 pipeline()
285 File "/usr/local/python3/lib/python3.5/site-packages/esrally/racecontrol.py", line 58, in call
286 step()
287 File "/usr/local/python3/lib/python3.5/site-packages/esrally/racecontrol.py", line 27, in call
288 self.command(self.ctx)
289 File "/usr/local/python3/lib/python3.5/site-packages/esrally/racecontrol.py", line 273, in
290 PipelineStep("build", ctx, lambda ctx: ctx.mechanic.prepare_candidate()),
291 File "/usr/local/python3/lib/python3.5/site-packages/esrally/mechanic/mechanic.py", line 28, in prepare_candidate
292 self._builder.build()
293 File "/usr/local/python3/lib/python3.5/site-packages/esrally/mechanic/builder.py", line 24, in build
294 self._exec("gradle.tasks.clean")
295 File "/usr/local/python3/lib/python3.5/site-packages/esrally/mechanic/builder.py", line 38, in _exec
296 gradle = self._config.opts("build", "gradle.bin")
297 File "/usr/local/python3/lib/python3.5/site-packages/esrally/config.py", line 114, in opts
298 raise ConfigError("No value for mandatory configuration: section='%s', key='%s'" % (section, key))
299 esrally.config.ConfigError: No value for mandatory configuration: section='build', key='gradle.bin'

And the gradle is installed just like:
root@laoyang .rally]# gradle -version

Gradle 2.14.1

Build time: 2016-07-18 06:38:37 UTC
Revision: d9e2113d9fb05a5caabba61798bdb8dfdca83719

Groovy: 2.4.4
Ant: Apache Ant(TM) version 1.9.6 compiled on June 29 2015
JVM: 1.8.0_91 (Oracle Corporation 25.91-b14)
OS: Linux 2.6.32-358.el6.x86_64 amd64

8

Hi @laoyang360,

yes, I know what's wrong. Rally does not know the exact path to the Gradle binary and hence cannot build from sources. Do you remember whether Rally asked you for it when you configured it (because it should do this)? Did you install it after you've configured Rally?

You can just add the following lines to /root/.rally/rally.ini:

[build]
gradle.bin = /usr/local/bin/gradle

I'm assuming above that gradle is installed in /usr/local/bin/gradle. Just update the path according to the output of which gradle on your system.

Please also ensure that your JAVA_HOME environment variable points to a JDK 8 installation so you can build Elasticsearch from sources.

Daniel

9

Hi,

I noticed two more things that will prevent you from building from sources:

  1. I still see the output fatal: Not a git repository (or any of the parent directories): .git. I guess you did not change the value for local.src.dir in your rally.ini.

  2. You have Gradle 2.14.1 installed. Elasticsearch currently requires exactly Gradle 2.13 (see this open ticket in the Elasticsearch project for more details)

I don't know whether you've seen it, but you are not required to build Elasticsearch from sources. You can also just benchmark a binary distribution. Say you want to benchmark Elasticsearch 2.3.0, then you'd enter:

esrally --pipeline=from-distribution --distribution-version=2.3.0

Rally will download the specified distribution from the official download site and benchmark it.

Daniel

10

Scratch that. I know where it's coming from. This is a bug Rally's own version detection. Nothing breaks as a consequence of that; it's just misleading and annoying. The fix will be included in the next release.

Daniel

11

tks.
@danielmitterdorfer:

I think I don't need to run from building sources, and I also installed elsticsearch V2.3.4 for long time, and the elaticsearch running ok.

I already revise my configuration, the total of my configuration as follows:
[root@laoyang .rally]# cat rally.ini
[meta]
config.version = 5

[system]
root.dir = /root/.rally/benchmarks
log.root.dir = logs
env.name = local

[source]
local.src.dir=/root/.rally/src
remote.repo.url=https://github.com/elastic/elasticsearch.git

[provisioning]
local.install.dir = install

[runtime]
java8.home = /opt/jdk1.8.0_91

[benchmarks]
local.dataset.cache = ${system:root.dir}/data

[reporting]
datastore.type = elaticsearch
datastore.host = 10.0.1.30
datastore.port = 9200
datastore.secure = False
datastore.user =
datastore.password =

[tracks]
default.url = https://github.com/elastic/rally-tracks

[build]
gradle.bin = /usr/bin/gradle

and my java and gradle version as follows:
[root@laoyang .rally]# echo $JAVA_HOME
/opt/jdk1.8.0_91
[root@laoyang .rally]# echo $GRADLE_HOME
/home/yangcy/gradle-2.14.1/bin
Now, I don't konw whether I just running esrally or "esrally --pipline=from-distribution --distribution-version=2.3.0" refer to what you said?

As my opinion, I think i already installed esrally, I should running "esrally --pipline=from-distribution --distribution-version=2.3.4".

But when I runnning:
[root@laoyang geonames]# esrally --pipeline=from-distribution --distribution-version=2.3.4
fatal: Not a git repository (or any of the parent directories): .git

    ____        ____
   / __ \____ _/ / /_  __
  / /_/ / __ `/ / / / / /
 / _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
                /____/


additional logs to /root/.rally/benchmarks/races/2016-08-05-07-58-31/local/logs/rally_out.log

fatal: Not a git repository (or any of the parent directories): .git
Downloading Elasticsearch 2.3.4 ...

FATAL: Cannot race

Reason: <urlopen error [Errno 101] Network is unreachable>

Please check the log file [/root/.rally/benchmarks/races/2016-08-05-07-58-31/local/logs/rally_out.log] for further details first.

If you think this is a bug, please file a report at https://github.com/elastic/rally/issues and include the log file for this race (/root/.rally/benchmarks/races/2016-08-05-07-58-31/local/logs/rally_out.log).

[root@laoyang ~]# cat /root/.rally/benchmarks/races/2016-08-05-07-58-31/local/logs/rally_out.log
2016-08-05 07:58:31,236 rally.main INFO Rally version [0.3.1]
2016-08-05 07:58:31,237 rally.main INFO Command line arguments: Namespace(advanced_config=False, car='defaults', challenge='append-no-conflicts', client_options='timeout:90,request_timeout:90', configuration_name=None, data_paths=None, distribution_repository='release', distribution_version='2.3.4', effective_start_date=datetime.datetime(2016, 8, 5, 7, 58, 31, 219811), offline=False, pipeline='from-distribution', preserve_install=False, quiet=False, report_file='', report_format='markdown', revision='current', rounds=1, subcommand=None, target_hosts='localhost:9200', telemetry='', track='geonames', user_tag='')
2016-08-05 07:58:31,276 rally.racecontrol INFO Resolved download URL [https://download.elasticsearch.org/elasticsearch/release/org/elasticsearch/distribution/tar/elasticsearch/2.3.4/elasticsearch-2.3.4.tar.gz] for version [2.3.4]
2016-08-05 07:58:31,277 rally.racecontrol INFO Downloading distribution for version [2.3.4].
2016-08-05 08:01:42,975 root ERROR A fatal error occurred while running subcommand [race].
Traceback (most recent call last):
  File "/usr/local/python3/lib/python3.5/urllib/request.py", line 1240, in do_open
    h.request(req.get_method(), req.selector, req.data, headers)
  File "/usr/local/python3/lib/python3.5/http/client.py", line 1083, in request
    self._send_request(method, url, body, headers)
  File "/usr/local/python3/lib/python3.5/http/client.py", line 1128, in _send_request
    self.endheaders(body)
  File "/usr/local/python3/lib/python3.5/http/client.py", line 1079, in endheaders
    self._send_output(message_body)
  File "/usr/local/python3/lib/python3.5/http/client.py", line 911, in _send_output
    self.send(msg)
  File "/usr/local/python3/lib/python3.5/http/client.py", line 854, in send
    self.connect()
  File "/usr/local/python3/lib/python3.5/http/client.py", line 1229, in connect
    super().connect()
  File "/usr/local/python3/lib/python3.5/http/client.py", line 826, in connect
    (self.host,self.port), self.timeout, self.source_address)
  File "/usr/local/python3/lib/python3.5/socket.py", line 711, in create_connection
    raise err
  File "/usr/local/python3/lib/python3.5/socket.py", line 702, in create_connection
    sock.connect(sa)
OSError: [Errno 101] Network is unreachable

During handling of the above exception, another exception occurred:
12

But, when I just running esrally, the follow information appered:
[root@laoyang ~]# esrally
fatal: Not a git repository (or any of the parent directories): .git

    ____        ____
   / __ \____ _/ / /_  __
  / /_/ / __ `/ / / / / /
 / _, _/ /_/ / / / /_/ /
/_/ |_|\__,_/_/_/\__, /
                /____/


Writing additional logs to /root/.rally/benchmarks/races/2016-08-05-08-22-47/local/logs/rally_out.log

fatal: Not a git repository (or any of the parent directories): .git
Preparing for race (might take a few moments) ...
  Building from sources ...

ERROR: Cannot race

Reason: Couldn't find a tar.gz distribution.

Please check the log file [/root/.rally/benchmarks/races/2016-08-05-08-22-47/local/logs/rally_out.log] for further details first.

If you need further help, please check Rally's docs at https://esrally.readthedocs.io or ask a question in the forum at https://discuss.elastic.co/c/elasticsearch.

If you think this is a bug, please file a report at https://github.com/elastic/rally/issues and include the log file for this race (/root/.rally/benchmarks/races/2016-08-05-08-22-47/local/logs/rally_out.log).

2016-08-05 08:22:50,124 rally.builder INFO Executing /usr/bin/gradle clean...
2016-08-05 08:22:54,425 rally.builder WARNING Executing '/usr/bin/gradle clean' failed
2016-08-05 08:22:54,426 rally.builder INFO Executing /usr/bin/gradle :distribution:tar:assemble...
2016-08-05 08:22:58,716 rally.builder WARNING Executing '/usr/bin/gradle :distribution:tar:assemble' failed
2016-08-05 08:22:58,718 root ERROR Cannot run subcommand [race].
Traceback (most recent call last):
  File "/usr/local/python3/lib/python3.5/site-packages/esrally/mechanic/builder.py", line 31, in add_binary_to_config
    binary = glob.glob("%s/distribution/tar/build/distributions/*.tar.gz" % src_dir)[0]
IndexError: list index out of range

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/python3/lib/python3.5/site-packages/esrally/rally.py", line 299, in dispatch_sub_command
    racecontrol.run(cfg)
  File "/usr/local/python3/lib/python3.5/site-packages/esrally/racecontrol.py", line 336, in run
    pipeline()
  File "/usr/local/python3/lib/python3.5/site-packages/esrally/racecontrol.py", line 58, in __call__
    step()
  File "/usr/local/python3/lib/python3.5/site-packages/esrally/racecontrol.py", line 27, in __call__
    self.command(self.ctx)
  File "/usr/local/python3/lib/python3.5/site-packages/esrally/racecontrol.py", line 274, in <lambda>
    PipelineStep("find-candidate", ctx, lambda ctx: ctx.mechanic.find_candidate()),
  File "/usr/local/python3/lib/python3.5/site-packages/esrally/mechanic/mechanic.py", line 31, in find_candidate
    self._builder.add_binary_to_config()
  File "/usr/local/python3/lib/python3.5/site-packages/esrally/mechanic/builder.py", line 33, in add_binary_to_config
    raise ImproperlyConfigured("Couldn't find a tar.gz distribution.")
esrally.exceptions.ImproperlyConfigured: Couldn't find a tar.gz distribution.

Until now, I think esrally is not very good for use. It's too difficult.

13

Hi @laoyang360,

ok, let's get this sorted out:

When you run esrally --pipline=from-distribution --distribution-version=2.3.4, Rally will download Elasticsearch from https://download.elasticsearch.org/elasticsearch/release/org/elasticsearch/distribution/tar/elasticsearch/2.3.4/elasticsearch-2.3.4.tar.gz

And according to the error message you could not reach this address from your computer as indicated by the network error: OSError: [Errno 101] Network is unreachable.

When you run just esrally, this is really just a convenience for the actual command that is run, which is: esrally --pipeline=from-sources --version=current. This means that Rally will build Elasticsearch from sources with Gradle. As I said you don't have the wrong version of Gradle installed (and Rally cannot really do anything about that).

Until now, I think esrally is not very good for use. It's too difficult.

I am sorry to hear that. I see your struggles and I see what I can do to provide better error messages. I see two problems in your case:

  • Your network connection seems to be quite unstable (slow? sometimes errors like "network unreachable"). I check whether we can make Rally more robust in this case and maybe retry downloads
  • Gradle is not installed in the correct version. Maybe we can add a check for that.

Do you have any other recommendations or suggestions?

Daniel

14

Hi, @danielmitterdorfer.
Thank your, May be I said not incorrect, please forgive me.

I download and install gradle 2.13 now,

the esrally is now running as follow:
2016-08-05 09:13:56,355 rally.track INFO [/root/.rally/benchmarks/data/geonames/documents.json.bz2] already exists locally. Skipping download.
2016-08-05 09:13:56,355 rally.metrics INFO Creating in-memory race store
2016-08-05 09:13:56,355 rally.supplier INFO Skip fetching sources
2016-08-05 09:13:56,367 rally.supplier INFO Specified revision [current] on command line results in git revision [046abdc
]
2016-08-05 09:13:56,369 rally.builder INFO Executing /usr/bin/gradle clean...

until now, there is no error apper, as the tips: " might take a few moments", I now wating.....

When I have new progress, I will tell you instantly.

thank you again sincerely.

15

Hi @laoyang360,

no worries. I hope it works now.

For the problems you've experienced I've opened https://github.com/elastic/rally/issues/125 and https://github.com/elastic/rally/issues/126.

I'll also soon release a fix for the nasty fatal: Not a git repository (or any of the parent directories): .git message.

Daniel

16

I have just released Rally 0.3.2 (see announcement) and the message should be gone after you've updated Rally.

17

Now I download and install the new version of esrally.
when running the new version of esrally, the follow information appers:
Because I wait for a long time, I interrupt the running.
[root@laoyang bin]# esrally

/ __ ____ / / / __
/ // / __ `/ / / / / /
/ , / // / / / // /
// ||_,///_, /
/____/

Writing additional logs to /root/.rally/benchmarks/races/2016-08-08-01-33-48/local/logs/rally_out.log

Preparing for race (might take a few moments) ...
Building from sources ...
^C
ERROR: Cannot race

Reason: Couldn't find a tar.gz distribution.

Please check the log file [/root/.rally/benchmarks/races/2016-08-08-01-33-48/local/logs/rally_out.log] for further details first.

If you need further help, please check Rally's docs at https://esrally.readthedocs.io or ask a question in the forum at https://discuss.elastic.co/c/elasticsearch.

If you think this is a bug, please file a report at https://github.com/elastic/rally/issues and include the log file for this race (/root/.rally/benchmarks/races/2016-08-08-01-33-48/local/logs/rally_out.log).
You have new mail in /var/spool/mail/root

The log errors:
265 }
266

267 2016-08-08 01:33:50,642 rally.track INFO [/root/.rally/benchmarks/data/geonames/documents.json.bz2] already exists locally. Skipping download.
268 2016-08-08 01:33:50,643 rally.metrics INFO Creating in-memory race store
269 2016-08-08 01:33:50,643 rally.supplier INFO Skip fetching sources
270 2016-08-08 01:33:50,653 rally.supplier INFO Specified revision [current] on command line results in git revision [046abdc
271 ]
272 2016-08-08 01:33:50,655 rally.builder INFO Executing /usr/bin/gradle clean...
273 2016-08-08 01:39:13,67 rally.builder INFO Executing /usr/bin/gradle :distribution:tar:assemble...
274 2016-08-08 01:56:31,781 rally.builder WARNING Executing '/usr/bin/gradle :distribution:tar:assemble' failed
275 2016-08-08 01:56:31,782 root ERROR Cannot run subcommand [race].
276 Traceback (most recent call last):
277 File "/usr/local/python3/lib/python3.5/site-packages/esrally/mechanic/builder.py", line 31, in add_binary_to_config
278 binary = glob.glob("%s/distribution/tar/build/distributions/*.tar.gz" % src_dir)[0]
279 IndexError: list index out of range
280

281 During handling of the above exception, another exception occurred:
282

283 Traceback (most recent call last):
284 File "/usr/local/python3/lib/python3.5/site-packages/esrally/rally.py", line 302, in dispatch_sub_command
285 racecontrol.run(cfg)
286 File "/usr/local/python3/lib/python3.5/site-packages/esrally/racecontrol.py", line 336, in run
287 pipeline()
288 File "/usr/local/python3/lib/python3.5/site-packages/esrally/racecontrol.py", line 58, in call
289 step()
290 File "/usr/local/python3/lib/python3.5/site-packages/esrally/racecontrol.py", line 27, in call
291 self.command(self.ctx)
292 File "/usr/local/python3/lib/python3.5/site-packages/esrally/racecontrol.py", line 274, in
293 PipelineStep("find-candidate", ctx, lambda ctx: ctx.mechanic.find_candidate()),
294 File "/usr/local/python3/lib/python3.5/site-packages/esrally/mechanic/mechanic.py", line 31, in find_candidate
295 self._builder.add_binary_to_config()
296 File "/usr/local/python3/lib/python3.5/site-packages/esrally/mechanic/builder.py", line 33, in add_binary_to_config
297 raise ImproperlyConfigured("Couldn't find a tar.gz distribution.")
298 esrally.exceptions.ImproperlyConfigured: Couldn't find a tar.gz distribution.

18

The second time I running esrally, The different follow information appers:
root@laoyang geonames]# esrally

Writing additional logs to /root/.rally/benchmarks/races/-02-14-14/local/logs/rally_out.log

Preparing for race (might take a few moments) ...
  Building from sources ...
Racing on track [geonames] and challenge [append-no-conflicts] with car [defaults]

ERROR: Cannot race

Reason: Could not start node 'rally-node0' within timeout period of 20.0 seconds. Please check the logs in '/root/.rally/benchmarks/races/-02-14-14/local/logs' for more details.

  If you need further help, please check Rally's docs at https://esrally.readthedocs.io or ask a question in the forum at https://discuss.elastic.co/c/elasticsearch.

If you think this is a bug, please file a report at https://github.com/elastic/rally/issues and include the log file for this race (/root/.rally/benchmarks/races/-02-14-14/local/logs/rally_out.log).

 02:14:16,590 rally.track INFO [/root/.rally/benchmarks/data/geonames/documents.json.bz2] already exists locally. Skipping download.
  268  02:14:16,591 rally.metrics INFO Creating in-memory race store
  269  02:14:16,591 rally.supplier INFO Skip fetching sources
  270  02:14:16,602 rally.supplier INFO Specified revision [current] on command line results in git revision [046abdc
  271 ]
  272  02:14:16,604 rally.builder INFO Executing /usr/bin/gradle clean...
  273  02:14:40,37 rally.builder INFO Executing /usr/bin/gradle :distribution:tar:assemble...
  274  02:16:14,557 rally.metrics INFO Creating in-memory metrics store
  275  02:16:14,557 rally.metrics INFO Opening metrics store for invocation=[20160808T021414Z], track=[geonames], challenge=[append-no-conflicts], car=[defaults]
  276  02:16:14,557 rally.provisioner INFO Preparing candidate locally in /root/.rally/benchmarks/races/-02-14-14/local/tracks/geonames/append-no-conflict s/install.
  277  02:16:14,558 rally.provisioner INFO Unzipping /root/.rally/src/distribution/tar/build/distributions/elasticsearch-5.0.0-alpha5-SNAPSHOT.tar.gz to /root/.rall y/benchmarks/races/-02-14-14/local/tracks/geonames/append-no-conflicts/install
  278  02:16:15,139 rally.provisioner INFO Using port [39200]
  279  02:16:15,140 rally.provisioner INFO Using data paths [['/root/.rally/benchmarks/races/-02-14-14/local/tracks/geonames/append-no-conflicts/install/e lasticsearch-5.0.0-alpha5-SNAPSHOT/data']]
  280  02:16:15,140 rally.launcher INFO Starting a cluster based on car [defaults] with [1] nodes.
  281  02:16:15,141 rally.launcher INFO ENV: {'HOME': '/root', 'MAIL': '/var/spool/mail/root', 'CVS_RSH': 'ssh', 'HISTCONTR
  282  02:16:15,141 rally.launcher INFO ES launch: ['bin/elasticsearch', '-Enode.name=rally-node0', '-Epath.logs=/root/.rally/benchmarks/races/-02-14-14/l ocal/logs/geonames/append-no-conflicts/server']
  283  02:16:16,132 rally.launcher INFO rally-node0: [ 10:16:16,131][WARN ][bootstrap ] [rally-node0] uncaught exception in thread [main]
  284  02:16:16,133 rally.launcher INFO rally-node0: org.elasticsearch.bootstrap.StartupError: java.lang.RuntimeException: can not run elasticsearch as root
  285 org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:105)
  286org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:96)
  287  02:16:16,134 rally.launcher INFO rally-node0: at org.elasticsearch.cli.SettingCommand.execute(SettingCommand.java:54)
  288  02:16:16,134 rally.launcher INFO rally-node0: at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:88)
  289  02:16:16,134 rally.launcher INFO rally-node0: at org.elasticsearch.cli.Command.main(Command.java:54)
  290  02:16:16,134 rally.launcher INFO rally-node0: at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:75)
  291 org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:68)
  292  02:16:16,135 rally.launcher INFO rally-node0: Caused by: java.lang.RuntimeException: can not run elasticsearch as root
  293  02:16:16,135 rally.launcher INFO rally-node0: at org.elasticsearch.bootstrap.Bootstrap.initializeNatives(Bootstrap.java:88)
  294  02:16:16,135 rally.launcher INFO rally-node0: at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:147)
  295  02:16:16,135 rally.launcher INFO rally-node0: at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:255)
  296  02:16:16,136 rally.launcher INFO rally-node0: at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:101)
  297  02:16:16,136 rally.launcher INFO rally-node0: ... 6 more
  298  02:16:35,148 rally.launcher ERROR Could not start node 'rally-node0' within timeout period of 20.0 seconds.
  299  02:16:35,148 root ERROR Cannot run subcommand [race].
19

Hi @laoyang360,

the startup log of Elasticsearch says:

284 02:16:16,133 rally.launcher INFO rally-node0: org.elasticsearch.bootstrap.StartupError: java.lang.RuntimeException: can not run elasticsearch as root

This means that you have to run the benchmarks as ordinary user. This is not related to Rally but a security measure of Elasticsearch itself.

Daniel

20

Hi @danielmitterdorfer

Now I running as user not root, running as another user named "elasticsearch".
When I running, I occur the same problem as Esrally - Couldn't find a tar.gz distribution when building from sources.

Your know, My gradle is 2.13 and my java is 1.8, just like:
[root@laoyang config]# echo $JAVA_HOME
/opt/jdk1.8.0_91
[root@laoyang config]# gradle
:help

Welcome to Gradle 2.13.

But I still have errors as follow:
2016-08-08-05-31-29/local/logs/rally_out.log).
[elasticsearch@laoyang bin]$ esrally

____        ____

/ __ ____ / / / __
/ // / __ `/ / / / / /
/ , / // / / / // /
// ||_,///_, /
/____/

Writing additional logs to /home/elasticsearch/.rally/benchmarks/races/2016-08-08-05-36-47/local/logs/rally_out.log

Preparing for race (might take a few moments) ...
Downloading sources from https://github.com/elastic/elasticsearch.git to /home/elasticsearch/.rally/benchmarks/src.
Cloning into '/home/elasticsearch/.rally/benchmarks/src'...
remote: Counting objects: 648049, done.
remote: Compressing objects: 100% (31/31), done.
remote: Total 648049 (delta 3), reused 0 (delta 0), pack-reused 648012
Receiving objects: 100% (648049/648049), 233.02 MiB | 145.00 KiB/s, done.
Resolving deltas: 100% (364756/364756), done.
Checking connectivity... done.
Building from sources ...

ERROR: Cannot race

Reason: Couldn't find a tar.gz distribution.

Please check the log file [/home/elasticsearch/.rally/benchmarks/races/2016-08-08-05-36-47/local/logs/rally_out.log] for further details first.

If you need further help, please check Rally's docs at https://esrally.readthedocs.io or ask a question in the forum at https://discuss.elastic.co/c/elasticsearch.

If you think this is a bug, please file a report at https://github.com/elastic/rally/issues and include the log file for this race (/home/elasticsearch/.rally/benchmarks/races/2016-08-08-05-36-47/local/logs/rally_out.log).

2016-08-08 06:20:19,438 rally.builder INFO Executing /usr/bin/gradle clean...
273 2016-08-08 06:30:03,814 rally.builder WARNING Executing '/usr/bin/gradle clean' failed
274 2016-08-08 06:30:03,816 rally.builder INFO Executing /usr/bin/gradle :distribution:tar:assemble...
275 2016-08-08 06:35:50,65 rally.builder WARNING Executing '/usr/bin/gradle :distribution:tar:assemble' failed
276 2016-08-08 06:35:50,67 root ERROR Cannot run subcommand [race].
277 Traceback (most recent call last):
278 File "/usr/local/python3/lib/python3.5/site-packages/esrally/mechanic/builder.py", line 31, in add_binary_to_config
279 binary = glob.glob("%s/distribution/tar/build/distributions/*.tar.gz" % src_dir)[0]
280 IndexError: list index out of range
281
282 During handling of the above exception, another exception occurred:
283
284 Traceback (most recent call last):
285 File "/usr/local/python3/lib/python3.5/site-packages/esrally/rally.py", line 302, in dispatch_sub_command
286 racecontrol.run(cfg)
287 File "/usr/local/python3/lib/python3.5/site-packages/esrally/racecontrol.py", line 336, in run
288 pipeline()
289 File "/usr/local/python3/lib/python3.5/site-packages/esrally/racecontrol.py", line 58, in call
290 step()
291 File "/usr/local/python3/lib/python3.5/site-packages/esrally/racecontrol.py", line 27, in call
292 self.command(self.ctx)
293 File "/usr/local/python3/lib/python3.5/site-packages/esrally/racecontrol.py", line 274, in
294 PipelineStep("find-candidate", ctx, lambda ctx: ctx.mechanic.find_candidate()),
295 File "/usr/local/python3/lib/python3.5/site-packages/esrally/mechanic/mechanic.py", line 31, in find_candidate
296 self._builder.add_binary_to_config()
297 File "/usr/local/python3/lib/python3.5/site-packages/esrally/mechanic/builder.py", line 33, in add_binary_to_config
298 raise ImproperlyConfigured("Couldn't find a tar.gz distribution.")
299 esrally.exceptions.ImproperlyConfigured: Couldn't find a tar.gz distribution

For summary: The above error is
ERROR: Cannot race
Reason: Couldn't find a tar.gz distribution.
esrally.exceptions.ImproperlyConfigured: Couldn't find a tar.gz distribution

I think the tips should tell me which .tar.gz is needed?

If your know the detail, please told me. Thanks very much.